Root cause of frozen 22KB growing MXF: the parent held a stray read-write
priming writer (FD 7) on the video FIFO while raw2bmx opened it, so raw2bmx
read a malformed/empty stream start and exited after the header (Duration:0).
Proven live on zampp3: removing the FD-7/8 priming + fd-watch loop and simply
starting ffmpeg before raw2bmx lets the blocking FIFO opens pair up naturally
(ffmpeg sole writer). True-1080p59.94 AVC-Intra 100 then grows monotonically
on disk and a mid-write snapshot decodes to its last frame (481 frames @ 8s).
ROOT CAUSE FOUND + verified. When growing video comes from fc_pipe, node pipes it to the bash orchestrator's stdin. ffmpeg ran as a backgrounded subshell merely inheriting fd 0 (0<&0). With a PIPE source (not the working file/FIFO case), that subshell was starved of the raw video -> filtergraph 'No filtered frames' -> empty mpeg2video -> raw2bmx broken pipe -> sidecar crash (write EPIPE). Reproduced exactly with 'fc_pipe | bash -c orchestrator'. Fix: save original stdin to FD 9 BEFORE the FIFO-priming fd games (exec 9<&0), point ffmpeg's fd0 at fd9 (0<&9), close 9 in raw2bmx + parent. Verified the live-equivalent path now produces a valid mpeg2video 4:2:2 yuv422p progressive 30000/1001 MXF matching the working Delta7 file. Also added EPIPE handlers so a broken pipe never crashes the sidecar.
Root cause of growing 'video empty / No filtered frames': when video comes from fc_pipe, node pipes it to the bash orchestrator's stdin (fd0). The ffmpeg subshell inherits fd0, BUT the parent bash kept fd0 open too (in its wait loop). Both held the read end of the same pipe, so the kernel split the raw video bytes between them and ffmpeg was starved -> zero decoded frames -> empty mpeg2video -> raw2bmx broken pipe. Manual 'fc_pipe | ffmpeg' worked because ffmpeg was the sole reader. Fix: parent execs 0</dev/null right after spawning ffmpeg, making ffmpeg the sole stdin reader. Verified the full pipeline (fc_pipe->ffmpeg mpeg2video 422->raw2bmx rdd9) produces a valid 1080p29.97 MXF matching the working Delta7 file.
ffprobe of the proven-working Delta7_20260603 file (Premiere opened it): mpeg2video 4:2:2 yuv422p 1920x1080 PROGRESSIVE 30000/1001 + 2x pcm_s16le. The 1080p59.94 source is frame-rate-halved to 1080p29.97 progressive (NOT interlaced, NOT 59.94p). Verified on-node: mpeg2video yuv422p -r 30000/1001 -> raw2bmx -t rdd9 --mpeg2lg_422p_hl_1080p -f 30000/1001 produces an MXF byte-identical in format to the working file (rc=0).
Verified on-node: libx264 high422 10-bit + x264 avcintra-class=100 -> raw2bmx -t op1a --avci100_1080p -f 60000/1001 produces a valid MXF (ffprobe: h264 High 4:2:2 Intra yuv422p10le 60000/1001). True 1080p59.94, openable in Premiere. CPU encode for now per demo deadline; NVENC h264 cannot do 4:2:2.
Today's codec churn (h264_nvenc/AVC-Intra/op1a) produced .mxf files Premiere could not open. Subagent diagnosis vs the old working files on the share proved the working format is XDCAM HD422 = mpeg2video 4:2:2 yuv422p, clip type rdd9, wrapped at 1080i59.94 (30000/1001). raw2bmx REJECTS MPEG-2 422 at 60000/1001, so a 1080p59.94 SDI feed must wrap as 1080i59.94. Reverted GROWING_VIDEO_ELEMENTARY_ARGS to mpeg2video, raw2bmx -t rdd9 --mpeg2lg_422p_hl_1080i -f 30000/1001, -f mpeg2video pipe, and rates() 59.94->30000/1001 with 1080 forced interlaced.
Using libx264 confirmed raw2bmx works with AVC-Intra parameters at 59.94. Switching back to hardware-accelerated h264_nvenc with matching parameters (High profile, all-intra GOP 1, yuv422p, aud) to keep CPU load low during 8-port burn tests.
XDCAM HD422 does not strictly support 1080p59.94, and ffmpeg/raw2bmx failed to negotiate the stream. Reverted to h264_nvenc (High profile, all-intra GOP 1, yuv420p) which raw2bmx can reliably wrap as OP1a (--avc_high) at 60000/1001. This restores NVENC hardware acceleration and Premiere edit-while-record compatibility.
Burn test: 5 assets errored during proxy with 'aborted'/'socket hang up'
during the master DOWNLOAD. The masters all exist in S3 (262-269MB) — it's
the connection-limited RustFS backend dropping streams when 8 jobs hammer it
at once. Two fixes:
1. downloadFromS3/uploadToS3 now retry transient failures (aborted, socket
hang up, ECONNRESET, timeout, 5xx, throttle) up to 5x with exponential
backoff, cleaning the partial file between download attempts. A single
mid-stream abort no longer errors the whole asset.
2. Reuse ONE shared S3 client instead of createS3Client()+client.destroy()
per call. The per-call destroy tore down the keep-alive agent's sockets
every time, so connection pooling never happened and each transfer opened
fresh connections — exactly what overwhelmed RustFS. A long-lived client
lets the keep-alive pool actually be reused.
NVIDIA_VISIBLE_DEVICES=1 was set but the sidecar still SAW /dev/nvidia0,1,2 and nvenc used GPU 0 — because capture sidecars run Privileged, which exposes every GPU device node regardless of NVIDIA_VISIBLE_DEVICES/DeviceRequests. Real fix: node-agent passes CAPTURE_GPU_INDEX to the sidecar and capture-manager adds ffmpeg '-gpu N' to the hevc_nvenc + h264_nvenc encoders, so each port's master+HLS encode is explicitly bound to its assigned L4. Spreads 8 ports across 3 cards.
NVIDIA_VISIBLE_DEVICES was set per-port correctly (0/1/2) but ALL encodes still landed on GPU 0 at 99%. Cause: HostConfig.DeviceRequests granted Count:-1 (all GPUs), which OVERRIDES NVIDIA_VISIBLE_DEVICES — the container saw all 3 cards and nvenc defaulted to device 0. Now build DeviceRequests with DeviceIDs:[chosenGpu] so each sidecar truly sees only its one L4.
_gpuCache was empty (probeGpusViaSmi container didn't populate it), so the count fell back to 1 → NVIDIA_VISIBLE_DEVICES=all again. Count /dev/nvidiaN nodes directly (visible in the privileged node-agent container, confirmed 3) — same method the heartbeat uses.
The node-agent image has no nvidia-smi binary, so the direct execFileSync
detect always failed → fell back to 1 GPU → NVIDIA_VISIBLE_DEVICES=all (the
exact bug we were fixing). Use the existing _gpuCache (populated at startup
by probeGpusViaSmi via a throwaway GPU container) for the count instead.
VIDEO FREEZE UNDER BURN (transient stall, self-recovers): all 8 capture
sidecars ran NVIDIA_VISIBLE_DEVICES=all with no -gpu selector, so ffmpeg
nvenc put every session (8 HEVC masters + 8 HLS = 16) on physical GPU 0
while the other two L4s sat idle. GPU 0 NVENC hit 86%, encode fell below
realtime, the framecache ring lapped → video froze → caught up → recovered.
Bridge verified smooth at 60fps throughout. FIX: node-agent now round-robins
each sidecar to a GPU by capture port (port % detected-GPU-count) via
NVIDIA_VISIBLE_DEVICES, honoring an explicit gpuUuid when set. Auto-detects
GPU count from nvidia-smi (override CAPTURE_GPU_COUNT). ~3 encoders/GPU now.
GOP PARSE: Number.parseFloat('60000/1001') returns 60000, making GOP 120000
(near open-GOP) instead of ~120. Added parseFps() to handle rational rates;
fixed hevcNvencArgs + buildHlsVideoArgs.
FILMSTRIP: RustFS object store intermittently returns NoSuchKey on GET for
keys that List/Head confirm exist, blanking the strip. Generation/queue/DB
all verified healthy (13/15 assets HAVE filmstrips). FIX: API now serves the
filmstrip JSON through itself with retry-on-NoSuchKey (succeeds within a
couple attempts) instead of handing the browser a signed URL — also closes
the S3 CORS gap. Frontend updated to consume the direct JSON.
Mount-health card showed ~31GB free for the growing SMB share when the NAS
actually has multi-TB. mam-api never mounts the CIFS share, so df on the
container's /growing path reported the local overlay filesystem. Now query
the share's true capacity via 'smbclient -c du' (no mount needed) using the
configured credentials; falls back to the local df + surfaces the probe
error if the share is unreachable. Added smbclient to the mam-api image.
Burn test: only 3 of 8 Deltacast ports reached 'receiving'; the rest stuck
'connecting' forever. Root cause was NOT the board (all 8 SDI ports lock +
feed framecache at 60fps — verified 8 live shm cursors). It was orphaned
standby sidecars squatting host ports 7441-7445: new sidecars hit
EADDRINUSE, got zero frames, and getStatus() reported 'connecting' forever.
freeCapturePort() pre-filtered the container list by .Image regex, but after
a wild-dragon-capture:latest rebuild the Docker list API degrades older
containers' .Image to a bare image ID — so the tag regex silently SKIPPED
the exact orphans holding the ports. Now we match by PORT env (survives
rebuilds) and guard with the inspected Config.Image (which keeps the tag),
so a port is always reclaimed before a new sidecar binds. This makes
enable/disable 'just work' across image rebuilds.
A/V REGRESSION (no audio + start stutter): capture-manager.js dropped the
-use_wallclock_as_timestamps 1 flag on the audio FIFO input (re-added by
d6b0b3a). Wallclock stamped audio by arrival time while video is CFR
frame-count, so audio ran 3-18% longer and master aresample padded seconds
of LEADING SILENCE → silent head, late video start, apparent 'no audio'.
Removing it restores the sample-count PTS baseline (8e5405c/55a72af):
audio shares the SDI clock domain, no drift, no pad.
GUI BUG A (elapsed showed 1hr+ on standby/just-started): frontend seeded
elapsed from recorder.started_at = the standby CONTAINER boot time (hours
old). Now seeds ONLY from the sidecar session duration (liveStatus.duration
when live.recording), shows nothing when idle. Backend /status now returns
session-scoped duration + recording flag, not container uptime.
GUI BUG B (false 'stopped' signal on idle ports): backend inferred signal
from container Running state (running->receiving, down->stopped) — so idle
standby ports with down sidecars showed red 'stopped'. Now signal comes
from the sidecar session (live.recording); standby = neutral 'idle', never
a false 'stopped'/'receiving'.
The growing_promote_after_seconds setting was stored but NEVER read — no
scanner existed, so growing clips only left the SMB share on a manual
right-click 'Move to S3'. This adds the missing automation:
- promotion-scanner.js: every 60s, finds pending_migration assets idle
(updated_at) longer than settings.growing_promote_after_seconds and
enqueues a promotion job. Idempotent (status guard + stable jobId) so
it's safe on every promotion worker. 12h default fallback.
- worker/index.js: starts the scanner on promotion-capable workers.
- Settings UI: the delay field is now 'Auto-promote to S3 after (hours)'
(converts hours<->seconds; storage stays seconds). Notes the manual
Library right-click 'Move to S3' option too.
Manual promotion (right-click Move to S3) and the safe HLS-segment live
thumbnail were already implemented and working.
Second half of the growing-never-engages bug. start() decided growing via the module-level const GROWING_ENABLED (captured false at standby boot) and referenced the now-removed GROWING_SMB_MOUNT const (ReferenceError, silently swallowed). Both made growingActive=false, so every growing record produced HEVC/S3 instead of XDCAM HD422 MXF. Now reads process.env.GROWING_ENABLED + growingSmbConfig().mount fresh at record start.
Root cause of stuck 'processing', failed deletes, and dead playback:
The mam-api proxies media (/video, /hls pipe the full S3 body through
Express), holding long-lived streaming sockets. With the SDK's default
http agents (no keep-alive, unbounded but unpooled) those streams starved
control-plane calls — DeleteObject and the proxy worker's master download
— which timed out (10s connectionTimeout) in bursts.
Fixes:
- mam-api S3 client: dedicated keep-alive http/https Agents (maxSockets 256)
+ requestTimeout raised 30s→300s so large master GETs finish.
- worker S3 client: previously had NO handler config at all (SDK defaults).
Added keep-alive agents + 600s requestTimeout so proxy/conform master
downloads (hundreds of MB) don't stall and leave assets in 'processing'.
Root cause of growing producing .mov instead of XDCAM HD422 .mxf:
mountGrowingShare() used module-level consts (GROWING_SMB_MOUNT etc.)
captured from process.env at IMPORT time. Standby capture containers boot
with these unset and receive the SMB mount/credentials per-session over
/capture/start (capture.js sets process.env right before start()). Because
the consts were already frozen empty, mountGrowingShare() saw no mount
source, returned false, and growing silently fell back to S3 streaming —
producing an HEVC .mov while the asset key said .mxf.
Fix: growingSmbConfig() reads process.env fresh at mount time. Also drop
the stale const guard in unmountGrowingShare().
Three fixes to restore growing-files (XDCAM HD422 MXF) recording:
1. capture-manager mountGrowingShare: pass username=/password= inline
instead of a credentials= file. TrueNAS SMB3 rejects the creds-file form
with EACCES (-13, 'cannot mount read-only') while the identical inline
creds mount fine. This was causing every growing record to silently fall
back to the HEVC/S3 path (producing .mov, not .mxf).
2. docker-compose capture: add cap_add SYS_ADMIN + DAC_READ_SEARCH and
apparmor:unconfined so mount.cifs can run inside the container.
3. storage /overview: wrap S3 HeadBucket/ListObjects probe in a 5s timeout
so the admin 'Mount health' card stops hanging on 'Probing…' forever
when S3 is slow.
Growing mode now shows an editable 'XDCAM HD422 bitrate (Mbps)' input (codec stays fixed to the growing MXF path). Default seeds to 50 Mbps for growing, 25 for GPU master. Backend already honored recording_video_bitrate via _buildGrowingOrchestrator -b:v/minrate/maxrate; this surfaces the control in the config modal.
- mam-api: dockerLogs() + demuxDockerStream() — the local container-log path
JSON.parsed Docker's raw multiplexed stream and always returned '(no logs)';
now strips stdcopy framing and returns readable text (tail configurable).
- web-ui: new Logs admin page — every container across every node grouped by
node in a left rail, live-follow log viewer with filter + copy on the right.
Reuses the now-working /cluster/containers/:node/:id/logs endpoint.
- web-ui: Containers screen now polls every 5s (was load-once) so the
cross-cluster view stays live without manual refresh.
- icons: add server + file glyphs (were referenced but missing -> blank).
- nav: Logs wired into the Admin sidebar section + routes + breadcrumbs.
THE root cause of 'container view only shows the primary': checkAgentAuth used
crypto.timingSafeEqual but 'crypto' was never imported (ES module). The call
threw ReferenceError, the try/catch swallowed it, _bearerEq returned false, so
EVERY bearer-token check on a node-agent failed. The primary's own containers
showed only because the local node-agent has no NODE_TOKEN (auth skipped).
Adding 'import crypto from crypto' makes token comparison work, so the primary
mam-api can now read containers + logs from every node.
/cluster/containers only returned the primary's containers: mam-api fanned out
to each node-agent with a single NODE_AGENT_TOKEN, but each node-agent only
accepted its own bound NODE_TOKEN, so remote nodes returned 401 and were
silently dropped (UI showed 'only zampp1').
node-agent now ALSO accepts a shared CLUSTER_READ_TOKEN (= mam-api's
NODE_AGENT_TOKEN) for the read-only container/log endpoints, so the aggregate
container view + per-container logs work across the whole cluster.
Removing wallclock made A/V length drift far worse (audio 11.8% long). The
known-clean config used wallclock + master aresample=async=1; the leading
silence is a standby backlog artifact addressed by the bridge live-edge flush +
record-start audio FIFO drain, not by changing the timestamp source.
The persistent ~2.5s of leading silence was the master aresample=async=1 PADDING
the audio to reconcile a PTS-origin mismatch: video PTS starts at frame 0
(-framerate), but -use_wallclock_as_timestamps stamped the first audio chunk at
its wall-clock arrival time (~2.5s after the ffmpeg graph opened). aresample
filled the gap with silence.
Drop wallclock: audio PTS now comes from the 48kHz sample count starting at 0 —
the same origin as video frame 0 — so the streams align with no pad. The bridge
already hands live audio (backlog flushed on attach), so no rate reference is
needed from wallclock.
The ~2.5s of leading silence at record start was the VHD audio slot QUEUE: while
the recorder is idle (no FIFO reader), the bridge blocks on open(O_WRONLY) but the
board keeps buffering audio slots. When the record ffmpeg attaches, the bridge
streamed that stale backlog first — heard as leading silence and pushing audio
out of alignment with the live video.
On each reader attach, drain slots that lock FAST (already-queued backlog) and
stop at the first lock that takes ~a frame period (= waiting on a live slot), so
the reader is handed the live edge, A/V aligned.
Root cause of 'silent first ~1s then clean' + ~0.5% audio-too-long: in standby
the bridge keeps filling the audio FIFO while the idle-preview consumes only
video, so when recording starts ffmpeg reads a ~0.5s backlog of stale audio,
AND the video-only pre-roll discards video frames the audio never had.
Fix: (1) skip the video-only pre-roll in standby (warm slot = no unstable
frames), (2) drain the audio FIFO non-blocking immediately before ffmpeg opens
it, so audio starts at the live edge aligned with the first real video frame.
The 16ch interleave in the deltacast bridge produced audio at HALF the correct
sample rate (measured 24224 vs 48000 samples/s/ch), which broke A/V sync and
pitch. Per the working baseline (audio was clean before the channel selector),
revert the bridge audio thread to the original single-group 2ch extraction and
the capture-manager audio input to -ac 2 + wallclock + aresample.
KEPT the good fixes: long-GOP HEVC for non-growing (NVENC realtime, no frame
drops) and GPU-only codec list. 16ch/channel-select is shelved for a separate,
properly-validated change.
The pre-roll drained only the video pipe (fc_pipe) while the audio FIFO kept
buffering, so ffmpeg read ~PRE_ROLL_SECONDS of surplus pre-roll audio — making
audio longer than video, which when synced compresses audio ~0.5% (pitch-up,
measured: 2591573 audio samples vs 2579395 expected for the video duration).
In standby the framecache slot is already warm (no unstable startup frames), so
the drain is unnecessary; skipping it lets ffmpeg open video and audio together
from the same instant. Cold on-demand spawns keep the brief drain.
- restore -use_wallclock_as_timestamps on audio input: without it ffmpeg's raw
s16le reader stalled the graph (NVENC idle at 9%, ~half frames dropped). With
it + long-GOP HEVC the encoder runs realtime and A/V length stays locked.
- remove all CPU codec options (prores*, dnxh*, libx264/265) from recorder UI;
GPU NVENC only (hevc_nvenc / h264_nvenc). 3x L4 cluster, no reason for CPU.
- GPU codec defaults in env builders + proxy default h264_nvenc.
The hevc_nvenc codec was hardcoded to all-intra (-force_key_frames expr:1), which
is ~4x the NVENC load. Applied to every recording it exceeded the L4's realtime
budget at 1080p59.94 10-bit -> fc_pipe dropped ~half the frames -> video came out
shorter than the (correct) audio -> A/V drift + pitch-up on playback.
Now all-intra is used ONLY when growing-files is on (where it's required for the
editable head). Normal recordings use efficient long-GOP HEVC (2s GOP, 2 B-frames)
which NVENC sustains in realtime with zero drops.
