WildDragonLLC/dragonflight

Fork 0

ZGaetano 67251a0dcd docs: add editor + growing file + feature expansion design spec

2026-05-18 19:14:00 -04:00

18 KiB

Raw Blame History

Wild Dragon MAM — Editor, Growing File Preview & Feature Expansion

Date: 2026-05-18
Status: Draft — awaiting user approval before implementation begins

Overview

Three interconnected areas, sequenced by priority:

NLE Editor (editor.html) — Premiere-style timeline editor with razor blade, in/out marking, and EDL export
Growing File Workflow — HLS live preview during SDI/SRT/RTMP capture, with rewind support
Feature Additions — Prioritized list of high-value additions (subclips, player improvements, bulk ops, waveform, etc.)

1. NLE Editor

1.1 Layout

A new page editor.html replaces the role of player.html for editorial work. player.html is kept as the lightweight browse/metadata view; the editor is the full-screen creative environment.

┌─ SOURCE MONITOR ──────┬─ PROGRAM MONITOR ──────┐   ← 40vh
│  [video preview]      │  [video preview]        │
│  TC: 00:00:00:00      │  TC: 00:00:00:00        │
│  [────scrub bar────]  │  [────scrub bar────]    │
│  [IN] [OUT] [Insert]  │  [J] [K] [L]  [⏩]     │
├─ MEDIA PANEL ─────────┴─ TIMELINE ─────────────┤   ← 60vh
│ [bin tree]    │  [V] [C] [H]  [──zoom──]        │
│ [asset list]  │  ruler: 00:00  00:05  00:10 …   │
│               │  V1 ░░░░[clip A]░░░░[clip B]░░  │
│               │  V2                              │
│               │  A1 ░░░░[clip A]░░░░[clip B]░░  │
│               │  A2                              │
└───────────────┴──────────────────────────────────┘

The editor is accessed from the library: each asset card gains an "Open in Editor" action (next to the existing delete button). Clicking opens editor.html?project=<id>&asset=<id> — loads the asset into the source monitor and creates or resumes the project's active sequence.

The sidebar nav gains an Editor link (between Library and Ingest).

1.2 Source Monitor

Displays the currently "loaded" clip (double-click asset in the media panel, or opened via ?asset= URL param)
Native <video> element with a custom transport bar (scrub slider, current-time / duration label, play/pause button)
Mark In: keyboard I or button — stores sourceIn in seconds
Mark Out: keyboard O or button — stores sourceOut in seconds
In/out range shown as a highlighted region on the scrub bar
Insert: drops the marked range into the timeline at the playhead, shifting downstream clips right
Overwrite: drops the marked range at the playhead, overwriting whatever is there
If no in/out marks are set, the full clip duration is used

1.3 Program Monitor

Plays back the timeline from the current playhead position
Virtual playback: a single <video> element cycles through clips — at each clip's source in-point, the video src is set to that clip's signed proxy URL and currentTime is seeked to sourceIn. When currentTime reaches sourceOut, the next clip loads. This is frame-close (not frame-perfect) and is acceptable for v1.
Timecode display: HH:MM:SS:FF derived from timeline position and the sequence frame rate
Transport shortcuts: Space (play/pause), J (reverse), K (stop), L (forward), ←/→ (± 1 frame), Home (jump to start)

1.4 Timeline

Ruler: Horizontal ruler showing timecode marks. Default scale: 100px/s. Adjustable via Ctrl + scroll wheel or a zoom slider.

Tracks: Four track rows — V1, V2 (video), A1, A2 (audio). Each track row has a fixed height (48px).

Clips: Absolutely positioned <div> elements within track rows.

left = timelineIn * scale
width = (timelineOut - timelineIn) * scale
Clip body shows: clip name (truncated), timecode range
If clip is wide enough (> 120px), shows the asset thumbnail centered

Playhead: Red vertical line spanning all tracks. Draggable. Clicking the ruler jumps the playhead.

Tools (keyboard shortcuts):

Tool	Key	Behavior
Select	`V`	Click to select (highlight), drag to move horizontally within track, drag left/right edge to trim source in/out
Razor	`C`	Click anywhere on a clip to split it into two clips at that exact frame
Hand	`H`	Click-drag to pan the timeline horizontally

Selection + editing:

Delete key removes selected clip(s)
Ctrl+Z / Ctrl+Shift+Z: undo / redo (state stored as a local history stack, max 50 steps)
Clips cannot overlap within the same track (razor and move operations enforce this)

1.5 Data Model

New migration: schema_patch_editor.sql

-- Sequences (named timelines within a project)
CREATE TABLE sequences (
  id          UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
  project_id  UUID NOT NULL REFERENCES projects ON DELETE CASCADE,
  name        TEXT NOT NULL DEFAULT 'Sequence 1',
  frame_rate  NUMERIC(6,3) NOT NULL DEFAULT 29.97,
  width       INTEGER NOT NULL DEFAULT 1920,
  height      INTEGER NOT NULL DEFAULT 1080,
  created_at  TIMESTAMPTZ DEFAULT NOW(),
  updated_at  TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX idx_sequences_project_id ON sequences(project_id);

-- Clips on a sequence timeline
CREATE TABLE sequence_clips (
  id                    UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
  sequence_id           UUID NOT NULL REFERENCES sequences ON DELETE CASCADE,
  asset_id              UUID NOT NULL REFERENCES assets ON DELETE CASCADE,
  track                 INTEGER NOT NULL DEFAULT 0,
  -- 0=V1, 1=V2, 100=A1, 101=A2
  timeline_in_frames    BIGINT NOT NULL,
  timeline_out_frames   BIGINT NOT NULL,
  source_in_frames      BIGINT NOT NULL DEFAULT 0,
  source_out_frames     BIGINT NOT NULL,
  created_at            TIMESTAMPTZ DEFAULT NOW(),
  updated_at            TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX idx_sequence_clips_sequence_id ON sequence_clips(sequence_id);

1.6 API Routes

All new routes live in services/mam-api/src/routes/sequences.js.

Method	Path	Description
GET	`/api/v1/sequences?project_id=X`	List sequences for project
POST	`/api/v1/sequences`	Create sequence (`{ project_id, name, frame_rate, width, height }`)
GET	`/api/v1/sequences/:id`	Get sequence + all clips (joined with asset thumbnail/name/proxy/fps)
PUT	`/api/v1/sequences/:id`	Update sequence metadata
DELETE	`/api/v1/sequences/:id`	Delete sequence and its clips
PUT	`/api/v1/sequences/:id/clips`	Replace entire clip array (full sync)
POST	`/api/v1/sequences/:id/export/edl`	Generate CMX3600 EDL string, return as `text/plain` download

Save strategy: The editor auto-saves (debounces 2s after any change). Ctrl+S triggers immediate save. The full clip array is sent on each save — simple and avoids fine-grained diff logic.

1.7 EDL Export

A new generateEDL(sequence, clips) function added to a shared utility (or inline in the export route) using the existing secondsToTimecode helper from worker/src/edl/parser.js (the logic is duplicated into the API since they're separate services — or promoted to a shared edl-utils module).

Output format: CMX3600 compatible with the existing conform.js worker.

TITLE: Sequence 1

001  clip_filename  V  C  00:00:00:00 00:00:05:00 00:00:00:00 00:00:05:00
002  other_clip     V  C  00:00:02:00 00:00:08:00 00:00:05:00 00:00:11:00

1.8 api.js Additions

// Sequences
getSequences(projectId)
createSequence(projectId, data)
getSequence(sequenceId)
updateSequence(sequenceId, data)
deleteSequence(sequenceId)
syncSequenceClips(sequenceId, clips)
exportSequenceEDL(sequenceId)  // triggers download

2. Growing File Workflow

2.1 Problem

During SDI capture, two fragmented MP4 streams are piped to S3 via multipart upload. Multipart uploads are invisible in S3 until CompleteMultipartUpload — so the proxy doesn't exist until recording stops. The UI has no way to preview an in-progress capture, and there's no way to rewind to review what was just captured.

For SRT/RTMP sources, there is no proxy at all until the BullMQ worker processes the file post-stop.

2.2 Solution: HLS Segmentation During Capture

Replace the pipe-to-S3 proxy step with local HLS segment writing during capture. After the recording stops, stitch the segments into a single MP4 and upload that to S3 as the proxy.

Why HLS:

Growing playlists are the standard browser mechanism for live video
Segments accumulate on disk — seeking back to the start of the capture (rewind) is natural
hls.js provides a tiny CDN-loadable library for Chrome/Firefox
FFmpeg's hls muxer is well-tested in production

FFmpeg proxy args change (SDI — currently spawned as a second process):

# Before (fragmented MP4 → pipe → S3):
ffmpeg [decklink input] \
  -c:v libx264 -preset fast -b:v 10M -c:a aac -b:a 192k \
  -movflags +frag_keyframe+empty_moov -f mp4 pipe:1

# After (HLS segments → local temp dir):
ffmpeg [decklink input] \
  -c:v libx264 -preset fast -b:v 10M -c:a aac -b:a 192k \
  -hls_time 2 \
  -hls_list_size 0 \
  -hls_flags append_list \
  -hls_segment_filename '/tmp/wd-hls/<sessionId>/seg%05d.ts' \
  /tmp/wd-hls/<sessionId>/index.m3u8

Disk budget: 2s segments at 10 Mbps ≈ 2.5 MB/segment. A 1-hour capture ≈ 9 GB temp disk. The capture host needs ≥ 10 GB free on the temp partition.

2.3 Capture Service Changes

capture-manager.js:

New HLS_SESSION_DIR env var (default: /tmp/wd-hls)
On start(): create /tmp/wd-hls/<sessionId>/, write HLS instead of piping to S3
proxyKey is still generated for the final S3 path; the HLS dir is an intermediate
liveUrl added to session state: /capture/live/<sessionId>/index.m3u8
On stop():
1. SIGINT the FFmpeg process (as now)
2. Wait for FFmpeg to flush the final segment and close the playlist
3. Run ffmpeg -i /tmp/wd-hls/<sessionId>/index.m3u8 -c copy <tmp>.mp4 to stitch
4. Upload stitched MP4 to S3 as proxy (using existing createUploadStream or direct upload)
5. rimraf(/tmp/wd-hls/<sessionId>/) to clean up
On startup: purge any /tmp/wd-hls/ dirs older than 24h (crash recovery)

New capture routes (services/capture/src/routes/):

GET /capture/live/:sessionId/index.m3u8   → stream the HLS playlist file
GET /capture/live/:sessionId/:segment     → stream a .ts segment file

Both routes check that the session is active (or was recently active) before serving.

Updated /capture/status response:

{
  "recording": true,
  "sessionId": "...",
  "liveUrl": "/capture/live/<sessionId>/index.m3u8",
  ...existing fields...
}

nginx proxy: Add location block to proxy /capture/live/ through to the capture service (same pattern as existing /capture/ pass-through).

2.4 SRT/RTMP Sources

Single FFmpeg process limitation: network streams can only be opened once. Use FFmpeg's multiple-output capability:

ffmpeg [srt/rtmp input] \
  -c:v prores_ks -profile:v 3 -c:a pcm_s24le \
  -movflags +frag_keyframe+empty_moov -f mov pipe:1 \   ← hires → S3 (unchanged)
  -c:v libx264 -preset fast -b:v 10M -c:a aac \
  -hls_time 2 -hls_list_size 0 -hls_flags append_list \
  -hls_segment_filename '/tmp/wd-hls/<sessionId>/seg%05d.ts' \
  /tmp/wd-hls/<sessionId>/index.m3u8                   ← proxy → local HLS

The hires upload stream continues unchanged. The proxy no longer goes to S3 during capture — it goes to HLS segments, then stitched on stop.

2.5 Live Preview UI (`capture.html`)

When the status poll returns recording: true with a liveUrl:

Show a collapsible Live Preview panel below the capture controls
Load hls.js from CDN: cdnjs.cloudflare.com/ajax/libs/hls.js/1.5.7/hls.min.js
new Hls() → hls.loadSource(status.liveUrl) → hls.attachMedia(videoEl)
video.play() on MANIFEST_PARSED event — plays at the live edge by default
"⏮ Rewind" button: video.currentTime = 0 — jumps to the start of the capture
Elapsed time counter ticking from startedAt

When recording stops: destroy the HLS instance, hide the preview panel. The asset appears in the library once the proxy upload and thumbnail job complete.

3. Feature Additions (Prioritized)

After Phase 1 (editor) and Phase 2 (growing file) are done, implement in this order:

P1 — Improved Player (`player.html` rebuild)

Current player.html uses the old CSS variable naming convention (--color-bg-tertiary, etc.) and doesn't match the redesigned system. Rebuild it to match index.html's design tokens and add:

Custom transport bar with scrub slider (replace browser defaults)
Timecode display HH:MM:SS:FF (using fps from asset metadata)
Frame-step buttons (± 1 frame)
J/K/L keyboard shortcuts
Inline rename: click display_name to edit in place
"Open in Editor" button linking to editor.html?asset=<id>

P2 — Subclips

In player.html, after marking in/out points:

"Create Subclip" button → POST /api/v1/assets with:

{ "parent_asset_id": "...", "subclip_in_ms": 12000, "subclip_out_ms": 45000, ... }

New columns on assets: parent_asset_id UUID REFERENCES assets, subclip_in_ms BIGINT, subclip_out_ms BIGINT
Library shows subclip cards with a "✂" badge; clicking opens the player pre-seeked to subclip_in_ms
Subclips use the same proxy S3 key as the parent; no extra transcoding needed

P3 — Multi-select & Bulk Ops

In index.html:

Shift-click or checkbox (appears on hover) to select multiple asset cards
Floating action bar appears at bottom when ≥1 card selected: Move to bin | Add tags | Delete
"Move to bin": slide panel with bin tree, moves selected assets via PATCH /assets/:id { bin_id }
"Add tags": text input, appends tags to all selected assets
Bulk delete: confirm modal, then soft-delete all

P4 — Waveform Display in Editor

During proxy generation (worker/src/workers/proxy.js):

After transcoding, run FFmpeg's volumedetect or astats filter to produce per-second peak arrays
Alternatively: ffmpeg -i proxy.mp4 -af astats=metadata=1:reset=1,ametadata=print:file=peaks.txt -f null -
Parse and store as { peaks: [0.2, 0.8, ...] } JSON → upload to waveforms/<assetId>.json in S3
New column on assets: waveform_s3_key TEXT
Editor timeline: for audio track clips, fetch the waveform JSON and render an SVG polyline within the clip block

P5 — Timecoded Comments

New table:

CREATE TABLE asset_comments (
  id              UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
  asset_id        UUID NOT NULL REFERENCES assets ON DELETE CASCADE,
  user_id         UUID NOT NULL REFERENCES users,
  timecode_seconds NUMERIC(10,3),
  body            TEXT NOT NULL,
  created_at      TIMESTAMPTZ DEFAULT NOW()
);

In player.html sidebar: "Comments" tab. Input field with "Post at [current TC]" button. Renders as a list of clickable entries that seek the video to that timecode.

P6 — Smart Bins

Extend bins table:

ALTER TABLE bins ADD COLUMN is_smart BOOLEAN DEFAULT false;
ALTER TABLE bins ADD COLUMN smart_query JSONB;

Smart bin smart_query shape: { "tags": ["interview"], "media_type": "video", "created_after": "2026-01-01" }. When loading a smart bin's assets, run a dynamically built query against the assets table. In the UI, smart bins show a ✦ icon in the bin tree and their contents cannot be manually modified.

P7 — Metadata Templates