- Growing-file masters (.mxf) are tagged smb on create (while live) and on
pending-migration; the tag swaps to s3 once promoted.
- Migrate-to-S3 (promote) now accepts assets stuck in live (sidecar post-stop
call never landed) in addition to pending_migration, guarded to .mxf SMB
masters only.
- Promotion queue added to the Jobs tab QUEUES so SMB->S3 migrations are
visible/trackable like other jobs.
- Library: SMB badge shows alongside LIVE for growing masters; Move to S3 shown
for any SMB-origin asset (live-stuck or pending_migration).
Verified: stuck-live 5.4GB master migrated SMB->S3, job tracked in Jobs tab,
tags swapped smb->s3.
Burn test: 5 assets errored during proxy with 'aborted'/'socket hang up'
during the master DOWNLOAD. The masters all exist in S3 (262-269MB) — it's
the connection-limited RustFS backend dropping streams when 8 jobs hammer it
at once. Two fixes:
1. downloadFromS3/uploadToS3 now retry transient failures (aborted, socket
hang up, ECONNRESET, timeout, 5xx, throttle) up to 5x with exponential
backoff, cleaning the partial file between download attempts. A single
mid-stream abort no longer errors the whole asset.
2. Reuse ONE shared S3 client instead of createS3Client()+client.destroy()
per call. The per-call destroy tore down the keep-alive agent's sockets
every time, so connection pooling never happened and each transfer opened
fresh connections — exactly what overwhelmed RustFS. A long-lived client
lets the keep-alive pool actually be reused.
The growing_promote_after_seconds setting was stored but NEVER read — no
scanner existed, so growing clips only left the SMB share on a manual
right-click 'Move to S3'. This adds the missing automation:
- promotion-scanner.js: every 60s, finds pending_migration assets idle
(updated_at) longer than settings.growing_promote_after_seconds and
enqueues a promotion job. Idempotent (status guard + stable jobId) so
it's safe on every promotion worker. 12h default fallback.
- worker/index.js: starts the scanner on promotion-capable workers.
- Settings UI: the delay field is now 'Auto-promote to S3 after (hours)'
(converts hours<->seconds; storage stays seconds). Notes the manual
Library right-click 'Move to S3' option too.
Manual promotion (right-click Move to S3) and the safe HLS-segment live
thumbnail were already implemented and working.
Root cause of stuck 'processing', failed deletes, and dead playback:
The mam-api proxies media (/video, /hls pipe the full S3 body through
Express), holding long-lived streaming sockets. With the SDK's default
http agents (no keep-alive, unbounded but unpooled) those streams starved
control-plane calls — DeleteObject and the proxy worker's master download
— which timed out (10s connectionTimeout) in bursts.
Fixes:
- mam-api S3 client: dedicated keep-alive http/https Agents (maxSockets 256)
+ requestTimeout raised 30s→300s so large master GETs finish.
- worker S3 client: previously had NO handler config at all (SDK defaults).
Added keep-alive agents + 600s requestTimeout so proxy/conform master
downloads (hundreds of MB) don't stall and leave assets in 'processing'.
The promotion worker promoted on mtime-idle (>=8s), but CIFS attribute caching
makes an actively-growing MXF look idle, so it grabbed the live file ~15s into
recording, uploaded it, flipped the asset live->ready, and unlinked it ("a
worker is stealing the file"). Gate promotion on the recorder's live status:
the growing asset's display_name is the recorder's current_session_id, so skip
promotion while a recorder with that session is status='recording'. Only
promote once recording has stopped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Add v2.2.3 to downloads (streaming write fix for large imports)
- Fix duration bug: worker now overwrites with ffprobe result instead of preserving capture wall-clock estimate
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Root cause: MXF OP1a writes its index/duration only in the footer partition
on finalize, so a growing MXF has no footer and VLC/Premiere/ffmpeg-strict
refuse it ("Unable to open file on disk"). Separately the proxy job pointed
at a .mov S3 key that never existed (promotion worker watched a local empty
disk, not the SMB share), so stop -> instant proxy failure.
Fix: growing master is now MPEG-TS (H.264 high422 all-intra + AAC), which is
readable from the first PAT/PMT while still growing (verified mid-write decode).
hiresKey derives from the actual produced extension. Capture skips finalize for
growing recorders (leaves asset live for promotion). Promotion worker CIFS-
mounts the same growing_smb share before scanning; worker image gets cifs-utils
and worker-p4 runs privileged (local /growing bind removed). /live-path uses .ts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- playout-stage: skip loudnorm pass 2 when measured_I=-inf (silent or
no-audio clip); fall back to plain AAC transcode so staging completes
instead of erroring out
- screens-home: add Playout tile; replace Premiere panel tile with
Downloads tile opening a combined modal (Premiere panel releases +
Dragon-ISO link to forge.wilddragon.net/WildDragonLLC/dragon-iso)
- screens-playout: add Delete channel button (visible only when stopped);
removes channel from list and selects next on confirm
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Stages playlist items from S3 to the shared CasparCG media volume. Pass 1
measures, pass 2 applies linear loudnorm (I=-23 LUFS, TP=-1 dBTP, LRA=11);
output is AAC 192k @ 48 kHz, video stream copied. Atomic rename on success
so CasparCG never sees a partial file. Per-item audio_normalized flag means
re-stages of the same asset skip the loudnorm pass.
Wired into worker/src/index.js behind WORKER_QUEUES=playout-stage so
capability-routed deploys can pin it to nodes that already have ffmpeg +
the media mount.
YouTube now packs AV1 inside .mp4, so the old `bv*[ext=mp4]` format selector
grabbed AV1 for the downloaded ORIGINAL. Premiere rejects AV1 on hi-res import
("unsupported file type"). The h264 proxy was fine, but the original wasn't.
Prefer avc1 (H.264) so the original is Premiere-native, falling back to the
previous any-mp4 behaviour only when no H.264 rendition exists. avc1 tops out
at 1080p on YouTube (above that is AV1/VP9 only), which is the universally
importable ceiling anyway.
Verified on a real URL: old selector -> av01 1080p; new selector -> avc1 1080p
(same resolution, now Premiere-native).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
YouTube imports were stuck in `ingesting` forever: after the capability-
routing split (fdec2e3), every worker's WORKER_QUEUES enumerated an explicit
list that omitted `import`, so index.js's `want('import')` was false on every
worker and nothing consumed the import queue. Add `import` to worker-p4
(heavy/primary worker; import is concurrency-1 and network-bound, one consumer
is enough).
Second defect that would have surfaced once routed: the baked yt-dlp came from
the distro package (Ubuntu 22.04 apt = 2022.04.08, ~4 years stale), which
current YouTube rejects. Pull the latest self-contained yt-dlp release binary
at image build instead (GPU image: yt-dlp_linux; alpine image: python zipapp).
Rebuild the worker image to refresh.
Verified on zampp1: import jobs drain ingesting -> processing -> ready,
failed=0, yt-dlp 2026.03.17.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
WORKER_QUEUES env lets a worker subscribe to a subset of queues. Deploy one GPU-pinned container per card: heavy encodes (proxy/conform/trim) on Tesla P4 (zampp1) + L4 (zampp2) via NVENC; light jobs (thumbnail/filmstrip) on the 2x Quadro P400 (zampp1). BullMQ competing-consumers distribute across nodes. RUN_PROMOTION gates the growing-files scanner to one worker. Each worker stamps WORKER_LABEL onto job data so the Jobs UI Node column shows which node/GPU ran each job. Redis/DB/S3 for the zampp2 worker come from its .env (pointed at zampp1).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ProRes / DNxHR conformed outputs are unplayable in the browser
(HTML5 video: MEDIA_ERR_SRC_NOT_SUPPORTED). The library was
referencing the ProRes original as the only source.
After the asset row is inserted, queue an H.264 proxy build the same
way services/mam-api/src/routes/assets.js does on ingest:
proxyQueue.add('generate', {
assetId,
inputKey: outputKey, // the conformed mov / mp4
outputKey: `proxies/${id}.mp4`,
});
The proxy worker writes the H.264 mp4, updates assets.proxy_s3_key,
and from then on /assets/:id/stream prefers the proxy over the
original. The library player can decode it natively.
Failure to enqueue is logged but doesn't fail the conform job — the
asset still exists and can have a proxy re-queued later.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The final concat-demux + encode step erred with:
[mp4] Could not find tag for codec prores in stream #0,
codec not currently supported in container
[out#0/mp4] Could not write header (incorrect codec parameters ?)
ProRes and DNxHR live in QuickTime (.mov), not MP4. The output path,
S3 key, and asset-row filename were all hardcoded to .mp4.
Pick the container from the codec:
prores / prores_hq / prores_4444 / dnxhr_hq → mov
h264 / h265 / anything else → mp4
outputExt is computed once at the top of the worker (before tmpfile
creation) and reused for the temp output, the S3 key
(jobs/<id>/conformed.<ext>), and the assets row's filename column.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
ffmpeg 8.x removed the `ocl` shortcut option from aresample (it was a
deprecated alias for out_chlayout). The per-segment trim+normalise call
errored immediately:
[fc#-1] Error applying option 'ocl' to filter 'aresample': Option not found
Split the chain: aresample handles the sample rate, aformat asserts +
auto-converts to stereo + fltp.
aresample=48000,aformat=channel_layouts=stereo:sample_fmts=fltp
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
ffmpeg 8.x's concat filter kept dying with the opaque
[fc#0] Error sending frames to consumers: Invalid argument
even after we locked fps + sample rate + pixel format + SAR in the
filter graph. Mixed sources (AV1+H.264, 23.98+60 fps, 44100+48000 Hz,
tv-range+unspecified-range pixel format) just don't survive the
concat filter cleanly in this build.
Switch to the more reliable 2-pass pattern:
1. At the trim step, re-encode each segment to a uniform intermediate
spec: libx264 ultrafast, 1920x1080 (letterboxed), yuv420p,
seqFps target rate, 48kHz stereo AAC. Per-segment ffmpeg.
2. At the concat step, use the concat *demuxer*. Because every input
now matches exactly, the demuxer is well-behaved. Transcode the
concatenated stream to the final target codec (ProRes 422 HQ etc).
Costs an extra intermediate encode (libx264 ultrafast ≈ realtime on
this hardware) but eliminates the filter-graph fragility on mixed-
source timelines, which is the workload that actually matters.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
After the demuxer → filter switch, concat still failed with
[fc#0] Error sending frames to consumers: Invalid argument
on Job 8. The filter graph normalised pixels (scale+pad+yuv420p) but
left the time-domain axes mixed:
segment-1: 23.98 fps video, 44100 Hz audio
segment-2: 60 fps video, 48000 Hz audio
segment-3: …
ffmpeg 8's concat filter requires identical frame rate + audio sample
rate + channel layout across inputs. Force them on each leg:
video: fps=<seqFps>, setpts=PTS-STARTPTS
audio: aresample=48000,
aformat=channel_layouts=stereo:sample_fmts=fltp,
asetpts=PTS-STARTPTS
setpts/asetpts re-zero each input's clock so concat's per-input PTS
window resets cleanly between segments.
Target fps comes from the sequence's frame_rate (rounded) — same axis
the sequence editor stores. Sample rate is pinned to 48000 (broadcast
standard) so the AAC encode is consistent.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
ffmpeg concat demuxer dies with "Error sending frames to consumers:
Invalid argument" when input segments don't share codec / pixel format
/ framerate / resolution. Mixed-source timelines hit this every time —
e.g. an AV1 clip + an H.264 clip going through the same concat.
Switch to the concat *filter*. It re-encodes through a filter graph
so disparate inputs are normalised inline. Each input is scaled to
1920x1080 with letterbox, format=yuv420p, audio resampled. concat=n=N
joins them into [outv]/[outa].
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three cooperating bugs left the rendered output silent and in the
wrong codec:
1. executor.js trimSegment used `-frames:v` with no audio mapping.
ffmpeg dropped the audio track on each segment before they reached
the concat step. Add `-c:a copy -shortest` so each segment carries
its original audio.
2. conform.js audioFlag was `audio === 'include' ? aac : -an`. The
panel's v2.2.1 defaults send `audio: 'broadcast'`, which didn't
match 'include' → `-an` explicitly stripped audio at the encode
step. Switch to the opposite default: only an explicit 'none' or
'off' disables audio; everything else gets AAC 320k @ 48kHz.
3. conform.js video codec map only matched `codec === 'prores'`. The
panel sends `'prores_hq'` (and the conform slide panel can send
`'prores_4444'` / `'dnxhr_hq'`). All of those fell through to
libx264 and silently rendered H.264 instead of the requested codec.
Add a real codec map with the right prores_ks profiles (3=HQ,
4=4444) and DNxHR. Skip -crf for ProRes since the profile encodes
quality.
The asset-row metadata's `codec` column is normalised the same way so
the new asset record matches what was actually written.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Panel had been sending xmeml with clipitem/name = the local Premiere
file path's basename (e.g. "dragonflight-Interstellar - Docking Scene
1080p IMAX HD.mp4"). The worker's old filename lookup ran
SELECT id, original_s3_key FROM assets WHERE filename = $1
which never matched, because the assets row's filename is the
original MAM ingest name without the "dragonflight-" prefix.
Fix: when job.data has sequenceId (always set by the conform endpoint
at routes/sequences.js:317), pull edits directly from sequence_clips,
which the panel already wrote with authoritative asset_id mappings on
push. We JOIN to assets for original_s3_key + filename and order by
(timeline_in_frames, track) so segment indices stay deterministic.
The XML is still parsed for sequence-level metadata (name, fps) when
provided, but its clipitems are no longer authoritative.
The legacy filename path (EDL input or fcpXml without sequenceId)
stays unchanged for backward compatibility.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
RustFS returns empty bodies for ranged GETs whose start offset is past
~5.9 MB on single-file proxy MP4s. HEAD reports correct size, full GET
(`bytes=0-`) works, but `bytes=8179166-` comes back 206 + correct
Content-Range header with zero bytes. Confirmed via direct S3 probe
against broadcastmgmt.cloud/dragonmam (see scratch tests).
Workaround in mam-api `GET /api/v1/assets/:id/video` until the proxy
worker emits HLS (planned v1.2.1):
- HEAD the object first to learn total size (also gives ETag /
Last-Modified for conditional requests).
- No-Range / unparseable-Range / pre-EOF requests \u2192 plain pipe.
- Parsed `bytes=N-M` requests below RUSTFS_RANGE_SAFE_START
(default 5_500_000) \u2192 direct ranged GET, RustFS handles fine.
- Anything reaching into the broken zone \u2192 stream from offset 0,
drop bytes below start, stop at end. Memory stays flat; extra
bandwidth = (end+1 - requested-size) per seek.
- Genuinely out-of-range \u2192 416 with Cache-Control: no-store so the
browser doesn't poison its cache.
Also stashes (not yet wired up) the HLS pieces we'll need for the
follow-up: `segmentToHls` ffmpeg helper + `uploadDirectoryToS3`
worker s3 helper. Harmless additions; not referenced by any code path
yet.
Confirmed against the affected asset (a72aaa03-...): bytes=0-100k +
50% +100k native pass-through; 70% +100k and near-EOF previously hung
the browser, now stream correctly via the stitched path.
Refs #143.
executor.js:
- transcodeVideo() now accepts videoMinRate, videoMaxRate, videoBufSize
- When set, passes -minrate/-maxrate/-bufsize to FFmpeg for ABR/VBR mode
- libx264 operates with per-scene quality variation within the envelope
proxy.js:
- Target average: 750k (gpu_bitrate_mbps=0.75)
- Min: 375k (50% of target), Max: 998k (~133%), Buffer: 2× max
- Gives effective range of ~500k-1M depending on scene complexity
- Log now shows VBR min-max-avg
- GPU fallback also passes VBR params
- Default videoBitrate changed from 10M to 750k in executor.js
- GET /assets/:id/stream now returns a signed S3 URL directly (4h TTL)
instead of pointing to the /video pipe endpoint. Browser streams
directly from S3 — no Node.js bottleneck, S3 handles range requests
natively for smooth seeking.
- GET /assets/:id/video now redirects (302) to a signed S3 URL.
Belt-and-suspenders: any code still calling /video gets redirected.
- proxy.js: default bitrate changed from 10Mbps to 1.5Mbps, audio
default from 192kbps to 128kbps. DB settings already updated to
1.5Mbps. Cuts proxy file size ~6x for the same quality content.
Existing proxies need re-generation at new bitrate.
Root causes found:
1. Scheduler crashing every 15s: assets table has no error_message column.
Fix: remove error_message from UPDATE in scheduler.js (#66 regression).
2. Clip freezing: client-side filmstrip seek loop runs on main thread,
seeks same proxy the player is streaming → both stall → freeze.
Fix: replace browser seek loop entirely with server-side FFmpeg worker.
3. No dedicated filmstrip worker: filmstrip was never pre-built server-side.
Changes:
- services/mam-api/src/db/migrations/018-add-filmstrip-s3-key.sql
Add filmstrip_s3_key TEXT column to assets table
- services/worker/src/workers/filmstrip.js (new)
BullMQ worker: downloads proxy, runs FFmpeg fps filter to extract
28 evenly-spaced JPEG frames, base64-encodes them, uploads JSON
array to S3 at filmstrips/<assetId>.json, stores key in DB
- services/worker/src/workers/thumbnail.js
Queue filmstrip job automatically after thumbnail completes
- services/worker/src/index.js
Register filmstrip worker (concurrency=2), export filmstripQueue
singleton, close it on SIGTERM
- services/mam-api/src/routes/assets.js
- filmstripQueue added
- POST /reprocess?type=filmstrip now supported
- GET /:id/filmstrip returns signed S3 URL for JSON frames
- services/mam-api/src/routes/jobs.js
filmstrip queue visible in Jobs UI
- services/web-ui/public/screens-asset.jsx
Replace browser seek loop with fetch of /assets/:id/filmstrip
→ fetch S3 JSON → render frames. Zero browser-side video seeking.
Right-click and Files tab re-generate via API endpoint.
track?.@_currentExplodedTrackIndex is invalid JS syntax — @ is not a
valid identifier character. Replaced with track?.['@_currentExplodedTrackIndex']
so the worker process no longer crashes on startup.
DELETE /jobs/:id was throwing "404 not found" when the operator tried to
cancel a running job. BullMQ refuses job.remove() while a job is in the
active state; the route caught that error and fell through to the
404 branch, which was misleading because the job actually exists — the
queue was just refusing to drop it from under the worker.
Fix:
- Detect 'active' state explicitly and call moveToFailed(err, '0', false)
first. Token '0' bypasses the per-worker lock check (the operator-side
cancel doesn't hold the worker lock). That transitions active -> failed
and frees the queue's concurrency slot.
- If moveToFailed itself fails (lock owned by a live worker), fall back
to job.discard() so at least the result is thrown away.
- If remove() then fails (stalled, broken state), drop the job's Redis
key directly via queue.client. Last-resort obliteration.
- Stop swallowing getJob() errors — if Redis is sad, surface it via
next(err) instead of returning a misleading 404.
- Return { cancelled: true } when the job was active, so the client
can show "Cancelled" rather than "Removed" in any future toast.
While here: thumbnail jobs now run with concurrency 4 by default
(proxy 2, conform 1, import 1 unchanged). Every queue defaulted to
concurrency 1 before, so a single stalled job blocked the entire queue.
All three are overridable via PROXY_CONCURRENCY / THUMBNAIL_CONCURRENCY
/ CONFORM_CONCURRENCY env vars for nodes with more headroom.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds Ingest → YouTube. UI takes a URL + project, API enqueues a BullMQ
"import" job, worker shells out to yt-dlp, lands the MP4 in S3 at the
same originals/{assetId}/... path uploads use, then hands off to the
existing proxy queue. Imported assets share one lifecycle with uploads
from that point on.
Worker container picks up yt-dlp + python3 (apk on alpine, apt on the
GPU variant). The new 'import' queue is registered in jobs.js so it
appears in the Jobs SSE stream and retry/delete work for free.
Spec: docs/superpowers/specs/2026-05-23-youtube-importer-design.md
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
path instead of failing the video transcode
Previously IMAGE_CODECS contained the raster ffprobe codec names ('png',
'mjpeg', 'jpeg', 'webp', 'gif', 'tiff', 'bmp', 'jpegls') but not 'svg'.
An SVG-as-asset (e.g. an architecture diagram dragged into a project) was
correctly tagged media_type='image' in the DB but ffprobe reported its
codec as 'svg', which fell through to the video branch, found
durationMs===null, and died with 'Empty or truncated source: codec=svg,
resolution=0x0'. That clogs the failed-jobs list with red rows that have
nothing to do with broken captures.
Two fixes here:
1) Add 'svg' to IMAGE_CODECS so the existing transcodeImage()/poster
path handles it.
2) Also bail to the poster path when the asset row itself says
media_type='image', even if ffprobe didn't return a codec name we
recognize (defensive — catches future formats like AVIF without
requiring an explicit catalog update).
Closes part of #13.
Proxy failures ("moov atom not found"):
- root cause: failed/aborted SRT/RTMP recordings still uploaded 0-byte
(or ftyp-only ~1KB) objects to S3, which ffmpeg can't probe
- worker proxy.js now bails on inputs < 4 KiB with a clear message
before handing the file to ffmpeg
- capture-manager.stop() returns framesReceived + empty flag
- capture shutdown handler skips POST /assets entirely on empty
sessions, instead calls new POST /assets/:id/mark-empty to flip
the pre-created live asset to 'error' with a note
Library asset right-click menu:
- new AssetContextMenu component on screens-library.jsx; right-click
any asset in grid or list view to open
- actions: Open, Rename, Move to bin (lists up to 10 bins), Remove
from bin, Copy asset ID, Delete permanently (hard=true)
- viewport-aware positioning (won't clip past window edges)
- dismisses on outside click / contextmenu / scroll
- Library now refreshes via /assets after mutations; normalizeAsset
exposed on window so the re-fetch shape matches boot
- ctx-menu styles in styles-rest.css
- Settings: drop AMPP tab, rename GPU/Transcoding → Proxy encoding
with explicit 'applied to every ingested file' wording, expose
CPU codec/preset options when GPU is off
- New Capture SDKs tab (Settings): upload Blackmagic / AJA / Deltacast
SDK archives (.zip / .tar.gz) staged to /sdk/<vendor>/ inside mam-api;
BMD is fully wired into the FFmpeg build pipeline, AJA + Deltacast
staging-only pending FFmpeg patches
- mam-api: new /api/v1/sdk routes (multer upload, extract, list, delete);
Dockerfile gets unzip+tar; docker-compose mounts /mnt/NVME/MAM/sdk:/sdk
- proxy worker now reads proxy-encoding settings from DB on every job,
builds args for libx264 / NVENC / VAAPI, falls back to libx264 on
hardware-encode failure
- settings GET /s3 falls back to S3_* env vars when DB is empty so the
UI reflects what's actually wired (fixes 'not configured' false alarm)
