ffprobe of the proven-working Delta7_20260603 file (Premiere opened it): mpeg2video 4:2:2 yuv422p 1920x1080 PROGRESSIVE 30000/1001 + 2x pcm_s16le. The 1080p59.94 source is frame-rate-halved to 1080p29.97 progressive (NOT interlaced, NOT 59.94p). Verified on-node: mpeg2video yuv422p -r 30000/1001 -> raw2bmx -t rdd9 --mpeg2lg_422p_hl_1080p -f 30000/1001 produces an MXF byte-identical in format to the working file (rc=0).
Verified on-node: libx264 high422 10-bit + x264 avcintra-class=100 -> raw2bmx -t op1a --avci100_1080p -f 60000/1001 produces a valid MXF (ffprobe: h264 High 4:2:2 Intra yuv422p10le 60000/1001). True 1080p59.94, openable in Premiere. CPU encode for now per demo deadline; NVENC h264 cannot do 4:2:2.
Today's codec churn (h264_nvenc/AVC-Intra/op1a) produced .mxf files Premiere could not open. Subagent diagnosis vs the old working files on the share proved the working format is XDCAM HD422 = mpeg2video 4:2:2 yuv422p, clip type rdd9, wrapped at 1080i59.94 (30000/1001). raw2bmx REJECTS MPEG-2 422 at 60000/1001, so a 1080p59.94 SDI feed must wrap as 1080i59.94. Reverted GROWING_VIDEO_ELEMENTARY_ARGS to mpeg2video, raw2bmx -t rdd9 --mpeg2lg_422p_hl_1080i -f 30000/1001, -f mpeg2video pipe, and rates() 59.94->30000/1001 with 1080 forced interlaced.
Using libx264 confirmed raw2bmx works with AVC-Intra parameters at 59.94. Switching back to hardware-accelerated h264_nvenc with matching parameters (High profile, all-intra GOP 1, yuv422p, aud) to keep CPU load low during 8-port burn tests.
XDCAM HD422 does not strictly support 1080p59.94, and ffmpeg/raw2bmx failed to negotiate the stream. Reverted to h264_nvenc (High profile, all-intra GOP 1, yuv420p) which raw2bmx can reliably wrap as OP1a (--avc_high) at 60000/1001. This restores NVENC hardware acceleration and Premiere edit-while-record compatibility.
NVIDIA_VISIBLE_DEVICES=1 was set but the sidecar still SAW /dev/nvidia0,1,2 and nvenc used GPU 0 — because capture sidecars run Privileged, which exposes every GPU device node regardless of NVIDIA_VISIBLE_DEVICES/DeviceRequests. Real fix: node-agent passes CAPTURE_GPU_INDEX to the sidecar and capture-manager adds ffmpeg '-gpu N' to the hevc_nvenc + h264_nvenc encoders, so each port's master+HLS encode is explicitly bound to its assigned L4. Spreads 8 ports across 3 cards.
VIDEO FREEZE UNDER BURN (transient stall, self-recovers): all 8 capture
sidecars ran NVIDIA_VISIBLE_DEVICES=all with no -gpu selector, so ffmpeg
nvenc put every session (8 HEVC masters + 8 HLS = 16) on physical GPU 0
while the other two L4s sat idle. GPU 0 NVENC hit 86%, encode fell below
realtime, the framecache ring lapped → video froze → caught up → recovered.
Bridge verified smooth at 60fps throughout. FIX: node-agent now round-robins
each sidecar to a GPU by capture port (port % detected-GPU-count) via
NVIDIA_VISIBLE_DEVICES, honoring an explicit gpuUuid when set. Auto-detects
GPU count from nvidia-smi (override CAPTURE_GPU_COUNT). ~3 encoders/GPU now.
GOP PARSE: Number.parseFloat('60000/1001') returns 60000, making GOP 120000
(near open-GOP) instead of ~120. Added parseFps() to handle rational rates;
fixed hevcNvencArgs + buildHlsVideoArgs.
FILMSTRIP: RustFS object store intermittently returns NoSuchKey on GET for
keys that List/Head confirm exist, blanking the strip. Generation/queue/DB
all verified healthy (13/15 assets HAVE filmstrips). FIX: API now serves the
filmstrip JSON through itself with retry-on-NoSuchKey (succeeds within a
couple attempts) instead of handing the browser a signed URL — also closes
the S3 CORS gap. Frontend updated to consume the direct JSON.
A/V REGRESSION (no audio + start stutter): capture-manager.js dropped the
-use_wallclock_as_timestamps 1 flag on the audio FIFO input (re-added by
d6b0b3a). Wallclock stamped audio by arrival time while video is CFR
frame-count, so audio ran 3-18% longer and master aresample padded seconds
of LEADING SILENCE → silent head, late video start, apparent 'no audio'.
Removing it restores the sample-count PTS baseline (8e5405c/55a72af):
audio shares the SDI clock domain, no drift, no pad.
GUI BUG A (elapsed showed 1hr+ on standby/just-started): frontend seeded
elapsed from recorder.started_at = the standby CONTAINER boot time (hours
old). Now seeds ONLY from the sidecar session duration (liveStatus.duration
when live.recording), shows nothing when idle. Backend /status now returns
session-scoped duration + recording flag, not container uptime.
GUI BUG B (false 'stopped' signal on idle ports): backend inferred signal
from container Running state (running->receiving, down->stopped) — so idle
standby ports with down sidecars showed red 'stopped'. Now signal comes
from the sidecar session (live.recording); standby = neutral 'idle', never
a false 'stopped'/'receiving'.
Second half of the growing-never-engages bug. start() decided growing via the module-level const GROWING_ENABLED (captured false at standby boot) and referenced the now-removed GROWING_SMB_MOUNT const (ReferenceError, silently swallowed). Both made growingActive=false, so every growing record produced HEVC/S3 instead of XDCAM HD422 MXF. Now reads process.env.GROWING_ENABLED + growingSmbConfig().mount fresh at record start.
Root cause of growing producing .mov instead of XDCAM HD422 .mxf:
mountGrowingShare() used module-level consts (GROWING_SMB_MOUNT etc.)
captured from process.env at IMPORT time. Standby capture containers boot
with these unset and receive the SMB mount/credentials per-session over
/capture/start (capture.js sets process.env right before start()). Because
the consts were already frozen empty, mountGrowingShare() saw no mount
source, returned false, and growing silently fell back to S3 streaming —
producing an HEVC .mov while the asset key said .mxf.
Fix: growingSmbConfig() reads process.env fresh at mount time. Also drop
the stale const guard in unmountGrowingShare().
Three fixes to restore growing-files (XDCAM HD422 MXF) recording:
1. capture-manager mountGrowingShare: pass username=/password= inline
instead of a credentials= file. TrueNAS SMB3 rejects the creds-file form
with EACCES (-13, 'cannot mount read-only') while the identical inline
creds mount fine. This was causing every growing record to silently fall
back to the HEVC/S3 path (producing .mov, not .mxf).
2. docker-compose capture: add cap_add SYS_ADMIN + DAC_READ_SEARCH and
apparmor:unconfined so mount.cifs can run inside the container.
3. storage /overview: wrap S3 HeadBucket/ListObjects probe in a 5s timeout
so the admin 'Mount health' card stops hanging on 'Probing…' forever
when S3 is slow.
Removing wallclock made A/V length drift far worse (audio 11.8% long). The
known-clean config used wallclock + master aresample=async=1; the leading
silence is a standby backlog artifact addressed by the bridge live-edge flush +
record-start audio FIFO drain, not by changing the timestamp source.
The persistent ~2.5s of leading silence was the master aresample=async=1 PADDING
the audio to reconcile a PTS-origin mismatch: video PTS starts at frame 0
(-framerate), but -use_wallclock_as_timestamps stamped the first audio chunk at
its wall-clock arrival time (~2.5s after the ffmpeg graph opened). aresample
filled the gap with silence.
Drop wallclock: audio PTS now comes from the 48kHz sample count starting at 0 —
the same origin as video frame 0 — so the streams align with no pad. The bridge
already hands live audio (backlog flushed on attach), so no rate reference is
needed from wallclock.
Root cause of 'silent first ~1s then clean' + ~0.5% audio-too-long: in standby
the bridge keeps filling the audio FIFO while the idle-preview consumes only
video, so when recording starts ffmpeg reads a ~0.5s backlog of stale audio,
AND the video-only pre-roll discards video frames the audio never had.
Fix: (1) skip the video-only pre-roll in standby (warm slot = no unstable
frames), (2) drain the audio FIFO non-blocking immediately before ffmpeg opens
it, so audio starts at the live edge aligned with the first real video frame.
The 16ch interleave in the deltacast bridge produced audio at HALF the correct
sample rate (measured 24224 vs 48000 samples/s/ch), which broke A/V sync and
pitch. Per the working baseline (audio was clean before the channel selector),
revert the bridge audio thread to the original single-group 2ch extraction and
the capture-manager audio input to -ac 2 + wallclock + aresample.
KEPT the good fixes: long-GOP HEVC for non-growing (NVENC realtime, no frame
drops) and GPU-only codec list. 16ch/channel-select is shelved for a separate,
properly-validated change.
The pre-roll drained only the video pipe (fc_pipe) while the audio FIFO kept
buffering, so ffmpeg read ~PRE_ROLL_SECONDS of surplus pre-roll audio — making
audio longer than video, which when synced compresses audio ~0.5% (pitch-up,
measured: 2591573 audio samples vs 2579395 expected for the video duration).
In standby the framecache slot is already warm (no unstable startup frames), so
the drain is unnecessary; skipping it lets ffmpeg open video and audio together
from the same instant. Cold on-demand spawns keep the brief drain.
- restore -use_wallclock_as_timestamps on audio input: without it ffmpeg's raw
s16le reader stalled the graph (NVENC idle at 9%, ~half frames dropped). With
it + long-GOP HEVC the encoder runs realtime and A/V length stays locked.
- remove all CPU codec options (prores*, dnxh*, libx264/265) from recorder UI;
GPU NVENC only (hevc_nvenc / h264_nvenc). 3x L4 cluster, no reason for CPU.
- GPU codec defaults in env builders + proxy default h264_nvenc.
The hevc_nvenc codec was hardcoded to all-intra (-force_key_frames expr:1), which
is ~4x the NVENC load. Applied to every recording it exceeded the L4's realtime
budget at 1080p59.94 10-bit -> fc_pipe dropped ~half the frames -> video came out
shorter than the (correct) audio -> A/V drift + pitch-up on playback.
Now all-intra is used ONLY when growing-files is on (where it's required for the
editable head). Normal recordings use efficient long-GOP HEVC (2s GOP, 2 B-frames)
which NVENC sustains in realtime with zero drops.
Removing -use_wallclock_as_timestamps on the SDI audio input. The bridge writes
SDI-clock-paced samples, so PTS from the 48kHz sample count shares the video's
clock domain and the audio length tracks the video length exactly. Wall-clock
timestamps made audio length = real elapsed time, which drifted ~1% longer than
the frame-count video when the encoder dipped under realtime (pitch-up).
- capture-manager: remove dead legacy deltacast FIFO video path (FC_SLOT_ID
is now always set by node-agent, framecache mandatory on all SDI nodes)
- node-agent: correct stale comment about legacy FIFO fallback
- onboard-node.sh: harden detect_sdi (device-node checks, not just lspci) and
persist COMPOSE_PROFILES so framecache survives every redeploy on SDI nodes
- remove committed capture.js.bak
Root cause of this session's outage: zampp3 came up without the capture
compose profile, so framecache never started; the bridge published to shm
with no consumer and recorders showed 'receiving' with no real capture.
The framecache ring delivers frame-accurate frames at exactly the SDI clock
rate. -use_wallclock_as_timestamps was wrong for this source — it stamped
frames by ffmpeg arrival time rather than capture time, causing the recorded
file to report wrong framerates (e.g. 56.06 instead of 59.94) and a
glitchy first second at startup (NVENC cold-start backlog bunched timestamps).
Fix: remove -use_wallclock_as_timestamps from the rawvideo (pipe:0) input
and rely on -framerate for correct CFR timestamps from frame 0.
Audio keeps its FIFO wallclock; aresample=async=1 on the master output
resamples audio to align with the CFR video PTS.
Reverts the local-temp+faststart approach from 549ca6c. Masters now stream
ffmpeg stdout directly to S3 via multipart upload — no local disk consumed
on the worker. Uses +frag_keyframe+empty_moov+default_base_moof which
Premiere Pro 25.x handles natively (to be confirmed separately).
Zero /tmp/capture files. Worker disk stays flat during recording.
- services/capture/src/capture-manager.js:
- Added 5s pre-roll delay for Deltacast, Blackmagic, and SDI capture paths.
- Activates when is present.
- Spawns the process immediately, drains/discards the unstable frames for 5 seconds, and then pipes the stdout of the to the actual process.
- Keeps the master output perfectly clean from frame 0.
- services/web-ui/public/screens-ingest.jsx:
- Removed currentFps framerate display from both recording and idle status states.
- services/framecache/src/net_ingest.c: new network ingest process
- Spawns ffmpeg to decode SRT/RTMP → raw UYVY422 stdout
- Reads decoded frames and writes into framecache slot via shm
- Registers slot with framecache HTTP API on startup
- Deregisters slot on clean exit (SIGTERM)
- Reconnect loop for listener mode (stays alive between sessions)
- --url, --slot-id, --fc-url, --width, --height, --fps-num/den,
--source-type, --listen, --listen-port, --stream-key args
- Emits format JSON to stderr on first frame
- services/framecache/CMakeLists.txt: add net_ingest target
- services/framecache/Dockerfile: copy net_ingest to runtime image
- services/node-agent/index.js:
- startNetIngest() / stopNetIngest(): lifecycle management per recorder
- Spawns net_ingest before sidecar start for srt/rtmp sourceTypes
- Injects FC_SLOT_ID=net-<containerId> into sidecar env
- Sets IpcMode=host for network sidecars using framecache
- Maps temp id → real containerId after container create
- stopNetIngest() called on sidecar stop
- NET_INGEST_BIN env var (default: docker exec framecache net_ingest)
- services/capture/src/capture-manager.js:
- _buildInputArgs srt/rtmp: framecache path when FC_SLOT_ID set
(spawns fc_pipe, uses pipe:0 rawvideo input — same as SDI path)
- Falls back to direct URL when FC_SLOT_ID not set (legacy path)
- audioMap: network via framecache uses '0🅰️0?' (video-only fc_pipe,
no audio FIFO — audio-in-shm is roadmap)
- HLS tee: sdiHlsDir covers network-via-framecache; legacy tee gated
on !FC_SLOT_ID to avoid duplicate HLS outputs
- fc_pipe piped to ffmpeg stdin for network framecache path
- docker-compose.worker.yml: FC_URL + NET_INGEST_BIN in node-agent env
fs.writeFile/fs.readFile/fs.stat are callback-based and don't return
Promises in the UXP sandbox. await on them resolves immediately, causing
race conditions where files aren't written before import.
Added _writeFile/_readFile/_stat wrappers that use fs.promises when
available and fall back to manual Promise wrapping otherwise.
Also bumped version to 2.2.3 to match web-ui data.jsx.
The live-thumbnail and manual /start,/stop sidecar->mam-api calls hit the CSRF
guard (403 missing X-Requested-With). Match the working pattern in index.js:
send Authorization: Bearer $MAM_API_TOKEN (= CAPTURE_TOKEN, injected by
recorders.js), which is CSRF-exempt. Falls back to the UI header only when no
token is set (dev). Fixes [livethumb] failed ... 403 — posters now persist.
🤖 Generated with Claude Code
Replace the HLS 'connecting…' player in the library with a real frame grabbed
from the start of the recording, while the recording is still live.
Flow:
- recorders.js already pre-creates the asset as status='live' + ASSET_ID env
- capture-manager.start() fires _publishLiveThumbnail() (non-blocking): polls
/live/<id> for the first seg-*.ts, extracts frame 0 via ffmpeg (scaled JPEG,
yuvj420p), uploads to S3 thumbnails/<id>.jpg, then POSTs the key to mam-api
- new mam-api POST /assets/:id/live-thumbnail sets thumbnail_s3_key on the still
-live row (status untouched); idempotent no-op once finalized
- visuals.jsx AssetThumb: for live assets, show the static poster once the key /
signed URL is available, else fall back to the live HLS preview. Pulsing LIVE
border kept either way
- POST /assets gains an optional status param (default 'processing'); 'live'
skips the proxy/thumbnail queue
- capture /stop route now finalizes the pre-created asset by id (guarded) instead
of POSTing a duplicate
🤖 Generated with Claude Code
ROOT CAUSE of 'connecting' hangs and intermittent port failures:
The DELTA-12G-e-h 8c is a bidirectional card. Without calling
VHD_SetBiDirCfg(board_index, VHD_BIDIR_80) before streaming, the
board remains in its default bi-dir config (likely 4RX/4TX) — so
RX stream opens fail with VHDERR_RESOURCEUNAVAILABLE on channels
configured as TX, causing random 'connecting' hangs per the SDK docs.
Per SDK Tools.cpp SetNbChannels() pattern:
1. Open temporary board handle
2. Check IS_BIDIR + channel counts
3. Call VHD_SetBiDirCfg(board_index, VHD_BIDIR_80) for 8ch bidir
4. Close temp handle, then open real board handle for streaming
Also add VHD_SetChannelProperty(VHD_CHANNEL_MODE_SDI) for ASI-type
channels per Sample_RX.cpp — required for 12G-ASI/3G-ASI channel
types to correctly detect incoming video standard.
🤖 Generated with Claude Code
The old architecture spawned one deltacast-capture per recorder port; each
called VHD_OpenBoardHandle, triggering a BufMngr.c:781 OOB fault in the
delta_x300 kernel driver whenever two opens raced.
Fix: a single deltacast-bridge daemon opens the board once, opens RX
streams for all requested ports concurrently, and writes each port's
video/audio to named FIFOs (/dev/shm/deltacast/video-<N>.fifo,
/dev/shm/deltacast/audio-<N>.fifo). Capture sidecars read from those
FIFOs directly — no board handle, no race, no flock.
Changes:
services/capture/deltacast-bridge/main.c
- Complete rewrite: --ports csv arg, board opened once, one
video+audio thread pair per port, FIFO paths per port, format
JSON emitted per port on signal lock, SIGTERM clean shutdown.
- flock/serialize logic removed (no longer needed).
- --port single-port compat alias retained.
services/capture/deltacast-bridge/CMakeLists.txt
- Rename target deltacast-capture -> deltacast-bridge.
- POST_BUILD symlink deltacast-capture -> deltacast-bridge for compat.
services/capture/src/capture-manager.js
- deltacast _buildInputArgs: remove bridge spawn; wait up to 30s
for FIFOs to exist (bridge may be starting); return rawvideo +
s16le FIFO inputArgs. bridgeProcess=null.
- audioMap: keyed on sourceType instead of bridgeProcess (both
inputs are always present for deltacast).
- Remove readFirstStderrLine helper (no longer needed).
- Remove bridgeProcess.stdout.pipe / processes.bridge stop signal.
services/node-agent/index.js
- Add import spawn for bridge daemon management.
- Add startDeltacastBridge / stopDeltacastBridge: host-process
lifecycle for the shared bridge, ref-counted by sidecar count.
- handleSidecarStart: on deltacast, increment counter + start bridge;
decrement on container create/start failure.
- handleSidecarStop: decrement counter; stop bridge when last sidecar.
- _containerSourceType map tracks containerId->sourceType for stop.
- Old acquireDcLock mutex retained but no longer called.
