356 lines
16 KiB
Markdown
356 lines
16 KiB
Markdown
# Wild Dragon MAM — Editor, Growing File Preview & Feature Expansion
|
||
**Date:** 2026-05-18
|
||
**Status:** APPROVED — ready for implementation planning
|
||
|
||
---
|
||
|
||
## Resolved Decisions
|
||
|
||
| Question | Decision |
|
||
|----------|----------|
|
||
| Save strategy | **Auto-save** (debounced 2s after any change) |
|
||
| Sequences per project | **Multiple named sequences** (like Premiere's project panel) |
|
||
| HLS temp disk | **3 TB available** — no constraint |
|
||
| Primary frame rate | **59.94 fps** |
|
||
| Program monitor fidelity | **Brief gap at clip boundaries is acceptable for v1** |
|
||
|
||
---
|
||
|
||
## Overview
|
||
|
||
Three sequenced phases:
|
||
|
||
1. **NLE Editor** (`editor.html`) — Premiere-style timeline editor with razor blade, in/out marking, and EDL export
|
||
2. **Growing File Workflow** — HLS live preview during SDI/SRT/RTMP capture, with rewind support
|
||
3. **Feature Additions** — Subclips, improved player, bulk ops, waveform, timecoded comments, smart bins, metadata templates
|
||
|
||
---
|
||
|
||
## 1. NLE Editor
|
||
|
||
### 1.1 Layout
|
||
|
||
A new page `editor.html`. `player.html` is kept as the lightweight browse/metadata view; the editor is the full-screen creative environment.
|
||
|
||
```
|
||
┌─ SOURCE MONITOR ──────┬─ PROGRAM MONITOR ──────┐ ← 40vh
|
||
│ [video preview] │ [video preview] │
|
||
│ TC: 00:00:00;00 │ TC: 00:00:00;00 │
|
||
│ [────scrub bar────] │ [────scrub bar────] │
|
||
│ [IN] [OUT] [Insert] │ [J] [K] [L] [⏩] │
|
||
├─ MEDIA PANEL ─────────┴─ TIMELINE ─────────────┤ ← 60vh
|
||
│ [sequence picker] │ [V] [C] [H] [zoom] │
|
||
│ [bin tree] │ ruler: 00:00 00:05 … │
|
||
│ [asset list] │ V1 ░░░░[clip A]░░░░ │
|
||
│ │ V2 │
|
||
│ │ A1 ░░░░[clip A]░░░░ │
|
||
│ │ A2 │
|
||
└───────────────────────┴─────────────────────────┘
|
||
```
|
||
|
||
Accessed from the library via an "Open in Editor" action on each asset card. Opens `editor.html?project=<id>&asset=<id>` — loads the asset into the source monitor and opens the project's most-recent sequence (or creates one named "Sequence 1" if none exists).
|
||
|
||
The sidebar nav gains an **Editor** link (between Library and Ingest).
|
||
|
||
### 1.2 Source Monitor
|
||
|
||
- Displays the currently loaded clip (double-click in media panel, or via `?asset=` param)
|
||
- Native `<video>` element with a **custom** transport bar: scrub slider, current-time / duration label, play/pause button
|
||
- **Mark In (`I`):** stores `sourceIn` as seconds; shown as left handle on the scrub bar
|
||
- **Mark Out (`O`):** stores `sourceOut` as seconds; shown as right handle on the scrub bar
|
||
- In/out range highlighted on scrub bar (accent color tint)
|
||
- **Insert:** drops marked range at timeline playhead, shifts downstream clips right
|
||
- **Overwrite:** drops marked range at timeline playhead, overwrites what's there
|
||
- No marks set → uses full clip duration
|
||
|
||
### 1.3 Program Monitor
|
||
|
||
- Plays the timeline from the current playhead position
|
||
- **Virtual playback:** single `<video>` element. On each clip start, set `src` = that clip's signed proxy URL and `currentTime` = `sourceIn`. When `currentTime` reaches `sourceOut`, load the next clip. Brief load gap at clip boundaries is acceptable for v1.
|
||
- Timecode display: `HH:MM:SS;FF` at **59.94 fps** (drop-frame notation, semicolon separator)
|
||
- Transport shortcuts: `Space` (play/pause), `J` (reverse), `K` (stop), `L` (forward), `←`/`→` (± 1 frame at 59.94), `Home` (jump to start)
|
||
|
||
### 1.4 Timeline
|
||
|
||
**Ruler:** Timecode-marked horizontal ruler. Default scale: 100px/s. Zoom: `Ctrl + scroll wheel` or zoom slider (range: 20px/s → 500px/s).
|
||
|
||
**Tracks:** V1, V2 (video), A1, A2 (audio). Each track row: 48px tall. Track header (40px wide, left): track label, lock toggle.
|
||
|
||
**Clips:** Absolutely positioned `<div>` elements within track rows.
|
||
- `left` = `timelineIn_seconds × scale`
|
||
- `width` = `(timelineOut - timelineIn)_seconds × scale`
|
||
- Shows: clip name (truncated), source TC range
|
||
- If width > 120px: asset thumbnail centered in clip body
|
||
|
||
**Playhead:** Red vertical line spanning all tracks. Draggable. Clicking the ruler jumps playhead.
|
||
|
||
**Tools:**
|
||
|
||
| Tool | Key | Behavior |
|
||
|------|-----|----------|
|
||
| Select | `V` | Click to select (accent border). Drag body to move horizontally. Drag left/right edge to trim source in/out. |
|
||
| Razor | `C` | Click on a clip → splits into two clips at that exact frame. |
|
||
| Hand | `H` | Click-drag to pan timeline horizontally. |
|
||
|
||
**Editing:**
|
||
- `Delete` / `Backspace`: remove selected clip(s)
|
||
- `Ctrl+Z` / `Ctrl+Shift+Z`: undo / redo (local history stack, max 50 steps)
|
||
- Clips cannot overlap within the same track (enforced on move/razor)
|
||
|
||
**Auto-save:** Debounced 2s after any timeline change. Visual indicator: subtle "Saving…" / "Saved" text in the timeline toolbar.
|
||
|
||
### 1.5 Data Model
|
||
|
||
**New migration: `schema_patch_editor.sql`**
|
||
|
||
```sql
|
||
CREATE TABLE sequences (
|
||
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
|
||
project_id UUID NOT NULL REFERENCES projects ON DELETE CASCADE,
|
||
name TEXT NOT NULL DEFAULT 'Sequence 1',
|
||
frame_rate NUMERIC(6,3) NOT NULL DEFAULT 59.94,
|
||
width INTEGER NOT NULL DEFAULT 1920,
|
||
height INTEGER NOT NULL DEFAULT 1080,
|
||
created_at TIMESTAMPTZ DEFAULT NOW(),
|
||
updated_at TIMESTAMPTZ DEFAULT NOW()
|
||
);
|
||
|
||
CREATE INDEX idx_sequences_project_id ON sequences(project_id);
|
||
|
||
CREATE TABLE sequence_clips (
|
||
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
|
||
sequence_id UUID NOT NULL REFERENCES sequences ON DELETE CASCADE,
|
||
asset_id UUID NOT NULL REFERENCES assets ON DELETE CASCADE,
|
||
track INTEGER NOT NULL DEFAULT 0,
|
||
-- track encoding: 0=V1, 1=V2, 100=A1, 101=A2
|
||
timeline_in_frames BIGINT NOT NULL,
|
||
timeline_out_frames BIGINT NOT NULL,
|
||
source_in_frames BIGINT NOT NULL DEFAULT 0,
|
||
source_out_frames BIGINT NOT NULL,
|
||
created_at TIMESTAMPTZ DEFAULT NOW(),
|
||
updated_at TIMESTAMPTZ DEFAULT NOW()
|
||
);
|
||
|
||
CREATE INDEX idx_sequence_clips_sequence_id ON sequence_clips(sequence_id);
|
||
```
|
||
|
||
**Frame math:** All frame counts use **59.94 fps** (= 60000/1001). Timecode display uses SMPTE drop-frame (`;` separator). Conversion helpers:
|
||
- `framesToSeconds(f)` = `f / 59.94`
|
||
- `secondsToFrames(s)` = `Math.round(s * 59.94)`
|
||
- Drop-frame TC calculation uses standard SMPTE DF algorithm
|
||
|
||
### 1.6 API Routes
|
||
|
||
New file: `services/mam-api/src/routes/sequences.js`
|
||
|
||
| Method | Path | Body / Params | Notes |
|
||
|--------|------|--------------|-------|
|
||
| GET | `/api/v1/sequences` | `?project_id=` | List sequences, ordered by `updated_at DESC` |
|
||
| POST | `/api/v1/sequences` | `{ project_id, name, frame_rate?, width?, height? }` | Create |
|
||
| GET | `/api/v1/sequences/:id` | — | Sequence + all clips joined with asset (`display_name`, `fps`, `duration_ms`, `proxy_s3_key`, `thumbnail_s3_key`) |
|
||
| PUT | `/api/v1/sequences/:id` | `{ name?, frame_rate?, width?, height? }` | Update metadata |
|
||
| DELETE | `/api/v1/sequences/:id` | — | Cascade deletes clips |
|
||
| PUT | `/api/v1/sequences/:id/clips` | `[ { asset_id, track, timeline_in_frames, timeline_out_frames, source_in_frames, source_out_frames } ]` | **Full replace** — deletes existing clips, inserts new array in one transaction |
|
||
| POST | `/api/v1/sequences/:id/export/edl` | — | Returns CMX3600 EDL as `text/plain; charset=utf-8`, `Content-Disposition: attachment` |
|
||
|
||
### 1.7 EDL Export
|
||
|
||
`generateEDL(sequenceName, clips, fps)` function produces CMX3600. Timecode math reuses the logic from `worker/src/edl/parser.js` (copied into a shared util or duplicated in the API route).
|
||
|
||
```
|
||
TITLE: My Sequence
|
||
|
||
001 clip_filename V C 00:00:00;00 00:00:05;00 00:00:00;00 00:00:05;00
|
||
002 other_clip V C 00:00:02;00 00:00:08;00 00:00:05;00 00:00:11;00
|
||
```
|
||
|
||
Conforms via the existing `conform.js` BullMQ worker unchanged.
|
||
|
||
### 1.8 `api.js` Additions
|
||
|
||
```js
|
||
getSequences(projectId)
|
||
createSequence(data)
|
||
getSequence(sequenceId)
|
||
updateSequence(sequenceId, data)
|
||
deleteSequence(sequenceId)
|
||
syncSequenceClips(sequenceId, clipsArray)
|
||
exportSequenceEDL(sequenceId) // triggers file download
|
||
```
|
||
|
||
---
|
||
|
||
## 2. Growing File Workflow
|
||
|
||
### 2.1 Problem
|
||
|
||
During SDI capture, proxy is piped to S3 via multipart upload — invisible until `CompleteMultipartUpload`. No live preview is possible. For SRT/RTMP, no proxy exists at all until the BullMQ worker runs post-stop.
|
||
|
||
### 2.2 Solution: HLS Segments During Capture
|
||
|
||
Write proxy as HLS segments to local disk during recording. Serve them live from the capture service. On stop: stitch segments → single MP4 → upload to S3 as proxy.
|
||
|
||
**Why HLS:** Growing playlists are the browser-native live video mechanism. Accumulated segments give free rewind. `hls.js` is a small CDN-loadable polyfill for Chrome/Firefox. FFmpeg's `hls` muxer is proven in production.
|
||
|
||
### 2.3 FFmpeg Args Change
|
||
|
||
**SDI (second FFmpeg process — proxy process):**
|
||
```bash
|
||
# Before: fragmented MP4 → pipe → S3
|
||
ffmpeg [decklink] -c:v libx264 -preset fast -b:v 10M -c:a aac -b:a 192k \
|
||
-movflags +frag_keyframe+empty_moov -f mp4 pipe:1
|
||
|
||
# After: HLS segments → local dir
|
||
ffmpeg [decklink] -c:v libx264 -preset fast -b:v 10M -c:a aac -b:a 192k \
|
||
-hls_time 2 -hls_list_size 0 -hls_flags append_list \
|
||
-hls_segment_filename '$HLS_DIR/<sessionId>/seg%05d.ts' \
|
||
$HLS_DIR/<sessionId>/index.m3u8
|
||
```
|
||
|
||
**SRT/RTMP (single FFmpeg process — two output pads):**
|
||
```bash
|
||
ffmpeg [srt/rtmp input] \
|
||
-c:v prores_ks -profile:v 3 -c:a pcm_s24le \
|
||
-movflags +frag_keyframe+empty_moov -f mov pipe:1 \
|
||
-c:v libx264 -preset fast -b:v 10M -c:a aac -b:a 192k \
|
||
-hls_time 2 -hls_list_size 0 -hls_flags append_list \
|
||
-hls_segment_filename '$HLS_DIR/<sessionId>/seg%05d.ts' \
|
||
$HLS_DIR/<sessionId>/index.m3u8
|
||
```
|
||
|
||
### 2.4 Capture Service Changes
|
||
|
||
**`capture-manager.js`:**
|
||
- New env var: `HLS_SESSION_DIR` (default: `/tmp/wd-hls`). 3 TB disk — no constraint.
|
||
- `start()`: `mkdir -p $HLS_SESSION_DIR/<sessionId>/`, spawn FFmpeg with HLS output. Add `liveUrl: /capture/live/<sessionId>/index.m3u8` to session state.
|
||
- `stop()`:
|
||
1. `SIGINT` FFmpeg (existing)
|
||
2. Wait for process exit (FFmpeg writes final segment + closes playlist before exiting)
|
||
3. `ffmpeg -i $HLS_DIR/<sessionId>/index.m3u8 -c copy <tmp>.mp4` (stitch)
|
||
4. Upload stitched MP4 to S3 as proxy key (existing `uploadToS3` helper)
|
||
5. `rm -rf $HLS_DIR/<sessionId>/`
|
||
- Startup: scan `$HLS_SESSION_DIR/` and delete dirs older than 24h
|
||
|
||
**New capture routes:**
|
||
```
|
||
GET /capture/live/:sessionId/index.m3u8 → serve HLS playlist file
|
||
GET /capture/live/:sessionId/:file → serve .ts segment files
|
||
```
|
||
Both routes: check session exists (active or recently stopped with dir still present), stream file from disk with correct `Content-Type` (`application/vnd.apple.mpegurl` / `video/mp2t`).
|
||
|
||
**Updated `/capture/status` response:**
|
||
```json
|
||
{
|
||
"recording": true,
|
||
"sessionId": "abc123",
|
||
"liveUrl": "/capture/live/abc123/index.m3u8",
|
||
"startedAt": "...",
|
||
"duration": 42,
|
||
...
|
||
}
|
||
```
|
||
|
||
**nginx:** Add location block for `/capture/live/` → proxy_pass to capture service (same pattern as existing `/capture/` block).
|
||
|
||
### 2.5 Live Preview in `capture.html`
|
||
|
||
When status poll returns `recording: true` with `liveUrl`:
|
||
1. Show collapsible **Live Preview** panel beneath capture controls
|
||
2. Load hls.js from CDN: `cdnjs.cloudflare.com/ajax/libs/hls.js/1.5.15/hls.min.js`
|
||
3. `new Hls()` → `hls.loadSource(status.liveUrl)` → `hls.attachMedia(videoEl)`
|
||
4. `videoEl.play()` on `MANIFEST_PARSED` (plays at live edge)
|
||
5. **⏮ Rewind** button: `videoEl.currentTime = 0`
|
||
6. Elapsed time counter from `status.startedAt`
|
||
|
||
On recording stop: `hls.destroy()`, hide preview panel. Asset appears in library after proxy upload + thumbnail job complete.
|
||
|
||
---
|
||
|
||
## 3. Feature Additions (Prioritized)
|
||
|
||
### P1 — Improved Player (`player.html` rebuild)
|
||
- Migrate from old CSS variables (`--color-bg-tertiary`) to current design tokens (`--bg-panel`, etc.)
|
||
- Replace browser default controls with custom transport bar: scrub slider, timecode display `HH:MM:SS;FF` at 59.94 fps, frame-step buttons, J/K/L shortcuts
|
||
- Inline rename: click `display_name` to edit in place → auto-save
|
||
- "Open in Editor" button → `editor.html?asset=<id>`
|
||
|
||
### P2 — Subclips
|
||
- Player shows In/Out markers → "Create Subclip" button
|
||
- `POST /api/v1/assets` with `{ parent_asset_id, subclip_in_ms, subclip_out_ms, display_name }`
|
||
- New columns: `parent_asset_id UUID REFERENCES assets`, `subclip_in_ms BIGINT`, `subclip_out_ms BIGINT`
|
||
- Library shows subclip cards with ✂ badge; player pre-seeks to `subclip_in_ms`
|
||
- Subclips use parent's proxy S3 key — no re-transcode
|
||
|
||
### P3 — Multi-select & Bulk Ops
|
||
- Shift-click or checkbox (visible on hover) to select multiple asset cards
|
||
- Floating action bar: **Move to bin | Add tags | Delete**
|
||
- Move to bin: slide panel with bin tree, `PATCH /assets/:id { bin_id }` for each
|
||
- Add tags: appends to all selected assets
|
||
- Bulk delete: confirm modal → soft-delete
|
||
|
||
### P4 — Waveform Display in Editor
|
||
- In proxy worker (`proxy.js`): after transcode, run FFmpeg `astats` filter to generate per-second peak arrays → store as `waveforms/<assetId>.json` in S3
|
||
- New `waveform_s3_key TEXT` column on `assets`
|
||
- Editor timeline: fetch waveform JSON for audio track clips, render as SVG polyline within clip block
|
||
|
||
### P5 — Timecoded Comments
|
||
```sql
|
||
CREATE TABLE asset_comments (
|
||
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
|
||
asset_id UUID NOT NULL REFERENCES assets ON DELETE CASCADE,
|
||
user_id UUID NOT NULL REFERENCES users,
|
||
timecode_seconds NUMERIC(10,3),
|
||
body TEXT NOT NULL,
|
||
created_at TIMESTAMPTZ DEFAULT NOW()
|
||
);
|
||
```
|
||
Player sidebar: "Comments" tab with "Post at current TC" button. Clickable entries seek video to that timecode.
|
||
|
||
### P6 — Smart Bins
|
||
```sql
|
||
ALTER TABLE bins ADD COLUMN is_smart BOOLEAN DEFAULT false;
|
||
ALTER TABLE bins ADD COLUMN smart_query JSONB;
|
||
-- smart_query shape: { "tags": ["interview"], "media_type": "video", "created_after": "2026-01-01" }
|
||
```
|
||
Smart bin assets: dynamically queried from `assets` table. Shows ✦ icon in bin tree.
|
||
|
||
### P7 — Metadata Templates
|
||
```sql
|
||
CREATE TABLE project_metadata_fields (
|
||
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
|
||
project_id UUID NOT NULL REFERENCES projects ON DELETE CASCADE,
|
||
field_key TEXT NOT NULL,
|
||
label TEXT NOT NULL,
|
||
field_type TEXT NOT NULL DEFAULT 'text',
|
||
options JSONB,
|
||
required BOOLEAN DEFAULT false,
|
||
sort_order INTEGER DEFAULT 0
|
||
);
|
||
ALTER TABLE assets ADD COLUMN metadata JSONB DEFAULT '{}';
|
||
```
|
||
|
||
---
|
||
|
||
## Implementation Sequencing
|
||
|
||
### Phase 1 — Editor
|
||
1. DB migration (`schema_patch_editor.sql`)
|
||
2. API (`sequences.js` route — CRUD + clip sync + EDL export)
|
||
3. `api.js` — sequence helpers
|
||
4. `editor.html` — 4-panel shell + CSS (sidebar nav update)
|
||
5. Timeline engine — ruler, playhead, track rows, clip rendering
|
||
6. Select tool — click-select, drag-move, drag-edge trim
|
||
7. Razor tool — click-to-split
|
||
8. Source monitor — video + transport + in/out marking + Insert/Overwrite
|
||
9. Program monitor — virtual playback + 59.94 drop-frame timecode
|
||
10. Auto-save (debounced 2s) + sequence picker in media panel
|
||
11. Library "Open in Editor" action + sidebar link
|
||
|
||
### Phase 2 — Growing File
|
||
1. Capture service: HLS output, `$HLS_SESSION_DIR` env var
|
||
2. Capture service: `/capture/live/:sessionId/` routes + nginx config
|
||
3. Capture service: on-stop stitch + S3 upload + cleanup
|
||
4. `capture.html`: live preview panel (hls.js), rewind button
|
||
|
||
### Phase 3 — Feature Additions
|
||
P1 → P2 → P3 → P4 → P5 → P6 → P7
|