Commit graph

1184 commits

Author SHA1 Message Date
9793ebea32 chore(panel): bump manifest to 2.2.4 (import-flow revert + conform fix) 2026-06-05 12:24:38 -04:00
e084e0a981 fix(panel): revert import-flow fs shim — broke proxy/hi-res import on UXP
v2.2.3 wrapped UXP's fs calls in a node-style `fs.promises || callback`
shim. UXP's require('fs') returns promise-based methods (not node
callback-style), so the fallback `fs.writeFile(path, data, cb)` path
never settles and proxy/hi-res import hung — import stopped working
entirely. Restore the v2.1.6 (2.2.2-shipped) direct `await fs.writeFile /
fs.stat / fs.readFile` form, which is verified working in Premiere.
Capture/recorder/deltacast untouched.
2026-06-05 12:24:28 -04:00
34809e9337 fix(panel): conform auto-resolves target project from clip assets
Conform was pushing clips to whatever project the operator picked in
#conform-proj-select. mam-api's PUT /sequences/:id/clips requires every clip
asset to belong to the sequence's project, so picking the "wrong" project (or
a timeline whose proxies were imported under a different project) produced a
silent HTTP 400 ("All clip assets must belong to the sequence's project") that
surfaced only as "clip push HTTP 400".

pushToMAM now looks up each matched asset's real project_id and:
  - if all matched assets share ONE project, conforms into THAT project
    (overriding the picked one) so the common "wrong project picked" case
    just works;
  - if the timeline genuinely mixes projects, throws a clear, actionable
    error naming the clips per project instead of a generic 400.

Pure editor/conform-path change. Does NOT touch capture, recorders, the
deltacast bridge, framecache, or node-agent.
2026-06-05 11:51:55 -04:00
7a08b90ce7 fix(panel): surface mam-api clip-push error body instead of bare HTTP 400
API.pushClips threw 'Clip push HTTP 400' and discarded the response body,
hiding the server's specific reason (e.g. "All clip assets must belong to
the sequence's project"). Now parse and include the {error} text so conform
failures are actionable. No behaviour change to the recorder/capture path.
2026-06-05 11:50:30 -04:00
OpenCode
9b4677cec7 refactor(capture): remove dead raw2bmx/AVC-Intra/HEVC growing code paths
Frame-coupled VC-3/DNxHD MXF (_buildGrowingVc3Mxf) is now the only growing
master path. Delete provably-unreferenced legacy builders and their constants,
and update stale comments to match the VC-3 reality. No live-path behavior
change: the removed stop() raw2bmx branch was gated on a hard-coded false, so
the kept SIGINT path is byte-identical to the previously-executed else branch.

Removed (all confirmed unreferenced repo-wide outside this file):
- _buildGrowingOrchestrator() + JSDoc (raw2bmx bash orchestrator)
- _buildGrowingHevcMov() + JSDoc (HEVC fragmented-MOV growing)
- deriveGrowingRaster() (raw2bmx raster-flag mapper)
- growingVideoElementaryArgs() + GROWING_AVCI_CLASS
- GROWING_DEFAULT_BITRATE, GROWING_PART_INTERVAL_FRAMES,
  GROWING_HEVC_DEFAULT_BITRATE, GROWING_VC3_CODECS (unused Set)
- isRaw2bmxGrowing dead branch in stop()
- stale raw2bmx/XDCAM HD422/AVC-Intra/frozen-picture comment blocks

Kept (live): _buildGrowingVc3Mxf, growingCodec, growingVc3Bitrate,
growingExtFor, GROWING_EXT, audioOffsetArgs (+AUDIO_OFFSET_MS), all
non-growing + network/SRT/RTMP/fc_pipe paths.

node --check passes.
2026-06-05 14:28:36 +00:00
OpenCode
b287ad08ef merge: frame-coupled embedded A/V capture (JOINED + AVI single-input)
Brings the working frame-coupled Deltacast capture to main:
- JOINED single-slot capture: video + SDI-embedded audio from the same VHD
  slot per frame (deltacast-bridge).
- Audio extraction fix: declare BOTH stereo channels for VHD_SlotExtractAudio
  (single-channel gave -91dB silence on this card/SDK).
- Single streaming AVI muxer in fc_pipe: one -f avi -i pipe:0 to ffmpeg,
  eliminating the two-raw-pipe ffmpeg startup deadlock and the separate audio
  FIFO transport entirely.

Verified end-to-end on live source: audio -15.5dB audible, video/audio packet
parity 1805/1805, identical durations (drift-free), 0 decode errors. No raw2bmx,
no separate audio FIFO. A/V sync confirmed by operator.
2026-06-05 14:20:58 +00:00
OpenCode
f5566cbb81 fix(bridge): JOINED audio needs BOTH stereo channels declared (fixes -91dB silence)
VHD_SlotExtractAudio on the JOINED slot returned -91dB silence when only
pAudioChannels[0] was declared. On this card/SDK declaring BOTH channel[0] and
channel[1] of the stereo pair makes the SDK land real PCM. Verified -13.3dB
audible against the live SDI source (was -91dB).
2026-06-05 14:12:23 +00:00
Zac Gaetano
97267aa857 fix(framecache): single-input streaming AVI muxer in fc_pipe (kills 2-pipe ffmpeg deadlock)
fc_pipe now muxes video + that frame's embedded audio into ONE streaming AVI
container on stdout, so ffmpeg reads a SINGLE input (-f avi -i pipe:0) instead
of a raw video pipe + a separate live audio FIFO. The two-live-pipe design
deadlocked ffmpeg (it stalled forever probing input 0); a single interleaved
stream removes that failure mode entirely.

fc_pipe.c:
  - New AVI mode (argv[3] == "--avi"/"avi"). Writes RIFF('AVI ') + LIST(hdrl)
    { avih + strl(vids: strh+BITMAPINFOHEADER UYVY 16bpp) + strl(auds:
    strh+WAVEFORMATEX PCM s16le 48k 2ch) } + LIST(movi) once, then per ring
    entry a '00dc' video chunk followed by a '01wb' audio chunk (LE sizes,
    even-pad). RIFF/movi sizes use the 0x7FFFFFFF streaming sentinel (pipe is
    unseekable); dwFlags has NO index bits. Frame-coupled by construction: both
    chunks come from the SAME ring entry in one read-loop iteration.
  - dwScale/dwRate = fps_den/fps_num (video) and nBlockAlign/nAvgBytesPerSec
    (audio). If a frame has audio_size 0, emits one frame-interval of silence
    (round(48000*fps_den/fps_num) samples) so the audio timeline tracks video
    and ffmpeg never starves on the audio demuxer.
  - Legacy raw video-only mode retained when no avi flag is given. The old
    split-stdout/audio-FIFO threaded path is removed (it was the deadlock).

fc_client.{h,c}:
  - Add fc_consumer_info() / fc_stream_info_t to expose the slot header's
    width/height/fps/audio params to fc_pipe for the AVI header.

capture-manager.js (_buildInputArgs deltacast/sdi framecache branch):
  - Spawn fc_pipe with "--avi" (no audio FIFO). Remove the mkfifo + audio-FIFO
    creation for this path.
  - inputArgs: ONE input  -thread_queue_size 512 -f avi [AUDIO_OFFSET_MS] -i pipe:0
    (was: -f rawvideo -i pipe:0  AND  -f s16le -ar 48000 -ac 2 -i <fifo>).
  - audioInputIndex 0, audioFifo null. Growing VC-3/HEVC builders already map
    [0:v] and audioMap 0🅰️0?; with one AVI input that resolves to 0:v / 0:a.

Validated on zampp3 against the LIVE deltacast-0-0 slot: fc_pipe --avi | ffmpeg
-f avi -i pipe:0 -> dnxhd/pcm_s24le MXF gives 360 video / 360 audio packets in
6.006s (no stall at 2 frames). A synthetic 1 kHz sine slot through the same
path yields mean_volume -9 dB / max -6 dB, proving the muxer carries real audio
end-to-end (the live SDI input currently carries no embedded audio, so the
bridge's silence fallback reads -91 dB — upstream of the muxer).
2026-06-05 14:06:35 +00:00
2f37119379 fix(framecache): frame-coupled audio — video+audio in ONE ring entry
Re-engineer the framecache so each video frame carries its own SDI-embedded
audio through ONE transport, eliminating the "audio ahead of video" offset at
the root: there is no longer a second independent audio buffer/FIFO that can
race ahead of video.

slot.h (FC_VERSION 1 -> 2):
  - Per ring entry data region is now [video frame_size][audio FC_MAX_AUDIO_BYTES].
  - fc_frame_t: the former _pad u32 is REUSED as audio_size (header still 24B).
  - Header gains audio_max_bytes / audio_rate / audio_channels (self-describing).
  - New fc_entry_stride()/fc_frame_audio() helpers; shm size includes audio.
  - Readers/writer check version == FC_VERSION and FAIL SAFE on mismatch so an
    old reader against a new writer (or vice-versa) refuses rather than misparses.

slot.c: populate audio header fields; add fc_slot_write_av(); version gate in open.
fc_client.[ch]: fc_frame_ref_t gains audio/audio_size; copy buffer holds
  video+audio; both copied from the SAME entry in one read -> frame-locked.
fc_pipe.c: now <slot_id> <wait_ms> <audio_fifo_path>; per ring entry writes
  video -> stdout AND that frame's audio -> the audio FIFO IN LOCKSTEP from one
  cursor read (no second buffer to drift). Auto-reattaches FIFO on EPIPE.

deltacast-bridge:
  - SILENT-AUDIO FIX: audio_extract_init now configures ONLY pAudioChannels[0]
    of group 0 as one stereo pair (Mode=STEREO, BufferFormat=AF_16, pData=buf),
    leaving pAudioChannels[1] ZEROED, exactly like Deltacast's own FFmpeg fork
    (libavdevice/videomaster_common.c init_audio_info). The prior JOINED code
    ALSO set channel[1].Mode/BufferFormat=STEREO/AF_16, declaring a second pair
    the signal does not carry -> VHD_SlotExtractAudio returned zero samples ->
    -91 dB silent audio. DataSize is (re)set to capacity before each extract.
  - VHD_SDI_SP_INTERFACE now set from the channel-detected interface
    (VHD_SDI_CP_INTERFACE) before StartStream, per the same fork — required for
    embedded-audio extraction on JOINED SDI streams.
  - fc_writer.[ch]: add fc_writer_write_av(); struct/stride bumped to v2.
  - video_thread (framecache path) extracts each frame's audio from the SAME
    locked JOINED slot and writes BOTH via fc_writer_write_av. Silence fallback
    at the source: a frame with no embedded audio gets one frame-interval of
    silence so the audio timeline length always equals the video timeline length.
  - The separate audio FIFO + audio_thread + apcm ring are retained ONLY for the
    legacy (-DLEGACY_FIFO / framecache-unreachable) fallback; on the primary
    framecache path the bridge no longer owns the audio FIFO.

capture-manager.js: deltacast/sdi framecache branch now CREATES the audio FIFO
  and passes its path to fc_pipe (argv[3]); ffmpeg keeps two raw inputs
  (rawvideo pipe:0 + s16le 48k input 1) but both are now fed frame-locked from
  the same ring entry. Stale-audio pre-flush retained as harmless safety.

All changes versioned; mismatched binaries refuse to attach (fail safe).
2026-06-05 12:46:22 +00:00
ZGaetano
80d8b15e8c fix(bridge): JOINED single-slot capture — embed audio with each video frame
Re-architect the Deltacast bridge from two independently-buffered VHD
streams (DISJOINED_VIDEO + a separate DISJOINED_ANC audio stream) to a
single VHD_SDI_STPROC_JOINED RX stream per port. Each locked slot now
carries both the video frame and that frame's SDI-embedded audio, so
audio is extracted (VHD_SlotExtractAudio) from the SAME slot the video
came from — eliminating the constant "audio ahead of video" offset at
its root instead of papering over it with --audio-delay-ms.

This mirrors Deltacast's own FFmpeg fork (libavdevice/videomaster_common.c):
open JOINED, then per frame LockSlot -> GetSlotBuffer(VIDEO) ->
SlotExtractAudio(same slot) -> Unlock.

Changes (main.c):
- main(): RX stream opened with VHD_SDI_STPROC_JOINED (was
  VHD_SDI_STPROC_DISJOINED_VIDEO). SDI interface still propagated so
  audio extraction yields real samples.
- video_thread(): becomes the single per-frame consumer. After locking
  each JOINED slot it extracts the embedded audio (audio_extract_slot)
  and pushes de-interleaved s16le stereo PCM into a new lock-free SPSC
  ring (ApcmRing), then writes the video frame to the framecache ring
  (or legacy video FIFO). Audio is extracted on BOTH the framecache and
  legacy-FIFO paths, before the video packing check so it is never
  dropped on a rare packing mismatch.
- audio_thread(): no longer opens any VHD stream. It is now purely the
  audio-FIFO sink: drains the ApcmRing into the named audio FIFO
  (ffmpeg input 1), flushes the ring to the live edge on reader attach,
  survives ffmpeg restarts (EPIPE -> reopen), and emits wall-clock-paced
  silence when the ring is empty (preserves the silence-fallback when the
  signal carries no embedded audio).
- Audio stays 48 kHz stereo s16le to match ffmpeg's expectations.
- --audio-delay-ms / FC_AUDIO_DELAY_MS retained but now unnecessary
  (should be left at 0); kept for compatibility / emergency tuning.

Compiles clean (only pre-existing %lu/ULONG format warnings in main()).
Not installed, not deployed — branch only.
2026-06-05 12:15:47 +00:00
OpenCode
e6f1313065 fix(promotion): heal dead CIFS mounts + retry file lookup; reset orphans on failure
The promotion worker mounts the growing SMB share, but a CIFS soft-mount can
stay mounted while DEAD (server dropped the connection) — every access then
returns ENOENT, so promotion fails Growing file not found and the asset is
stranded in processing (recurring stuck-migration bug). Fixes:
- ensureGrowingShareMounted now PROBES the mount with a readdir; if dead, lazy-
  unmounts and remounts fresh (was: returned early if anything was mounted).
- file lookup retries for ~20s (CIFS attribute-cache lag on a freshly written
  master), remounting between attempts.
- on any promotion failure, the asset is reset (pending_migration if the file is
  present, else error) instead of being left in processing forever.
2026-06-05 11:45:10 +00:00
OpenCode
641b033bf4 feat: per-recorder audio_offset_ms dial for A/V alignment
Adds a per-recorder audio offset (ms) control, applied as an ffmpeg -itsoffset
on the audio input: positive delays audio (fixes audio-ahead), negative
advances, 0 = none. Flows DB (migration 037) -> mam-api (RECORDER_FIELDS +
env AUDIO_OFFSET_MS + start body) -> capture.js (sets process.env per session)
-> capture-manager audioOffsetArgs() -> ffmpeg. UI: number field in the
recorder config modal. Verified end-to-end (setting 120 -> -itsoffset 0.1200 on
the live ffmpeg). Default 0, clamped +/-1000ms, non-destructive.

Note: this is an interim trim control; the root-cause A/V fix (Deltacast JOINED
single-slot embedded-audio extraction) is tracked separately.
2026-06-05 11:24:11 +00:00
OpenCode
feeab99a36 feat(bridge): --audio-delay-ms knob for deterministic A/V alignment
The deltacast bridge captures audio (DISJOINED_ANC VHD stream -> FIFO) and
video (VHD stream -> framecache ring -> fc_pipe -> ffmpeg) on separate paths
with independent buffering. The video path is buffered deeper, so audio reaches
the muxer AHEAD of its matching video frame (user-confirmed: audio ahead of
video on lip-sync). --audio-delay-ms / FC_AUDIO_DELAY_MS prepends N ms of real
PCM silence to the audio stream once per reader-attach, shifting the audio
timeline N ms later to re-align. One value, all 8 ports, drift-free (ffmpeg
derives audio PTS from sample count). Default 0 = unchanged (non-destructive).
Operator tunes once against a lip-sync reference. Also fixes promotion worker
to reset orphaned processing assets on failure (was stuck forever).
2026-06-05 11:00:15 +00:00
OpenCode
3b3e7edade feat(ui): segmented 90/220 toggle for growing codec (cleaner than dropdown)
Replace the VC-3 growing-codec dropdown with a clean two-button segmented
toggle (90 Mbps / 220 Mbps) in both the recorder config modal and the new-
recorder modal. Shows the bitrate + a one-line tradeoff (Lighter/default vs
Highest quality) so operators can swap quality at a glance. Removed the now-
redundant growing bitrate text.
2026-06-05 05:29:26 +00:00
OpenCode
c40de38c45 feat(capture): AUDIO_OFFSET_MS knob for fixed A/V alignment trim
The deltacast bridge captures audio and video on separate VHD streams; any
constant capture-path latency difference shows as a fixed A/V offset (e.g.
audio slightly ahead of video) even though stream lengths stay locked (no
drift, verified ~1 frame over 461s). AUDIO_OFFSET_MS applies an -itsoffset on
the SDI/Deltacast audio input only: positive DELAYS audio (audio-ahead case),
negative advances it. Default 0 = no change, fully non-destructive, clamped
to +/-1000ms. Lets an operator dial out residual offset with a lipsync loop
without a code change.
2026-06-05 05:27:27 +00:00
OpenCode
e64281c9fd fix(worker): proxy + conform handle VC-3/DNxHD MXF and ProRes correctly
proxy.js: the empty-source guard tripped on growing VC-3/DNxHD MXF masters,
which carry a valid decodable video stream but report format.duration=N/A
(durationMs=null). Only bail when there is ALSO no resolution (the true
aborted-capture signature), so MXF masters transcode instead of erroring.
Verified: burn-test growing masters now migrate SMB->S3 AND generate proxies
(assets dc0/dc2/dc6 ready with proxy, tags s3).

conform.js: pin codec-correct pixel format per preset (ProRes HQ yuv422p10le,
ProRes 4444 yuva444p10le, DNxHR HQ yuv422p); only emit -preset/-crf for
libx264/libx265 (dnxhd/prores drive quality via profile); and use the
codec-correct output extension in the error handler (was hard-coded .mp4, so a
failed ProRes/DNxHR conform never flipped to error and spun in processing
forever - the broadcast-vs-web asymmetry). All presets verified rc=0.
2026-06-05 05:23:16 +00:00
OpenCode
5c07b4e8b1 feat(assets): SMB tag + always-available S3 migrate for growing masters
- Growing-file masters (.mxf) are tagged smb on create (while live) and on
  pending-migration; the tag swaps to s3 once promoted.
- Migrate-to-S3 (promote) now accepts assets stuck in live (sidecar post-stop
  call never landed) in addition to pending_migration, guarded to .mxf SMB
  masters only.
- Promotion queue added to the Jobs tab QUEUES so SMB->S3 migrations are
  visible/trackable like other jobs.
- Library: SMB badge shows alongside LIVE for growing masters; Move to S3 shown
  for any SMB-origin asset (live-stuck or pending_migration).

Verified: stuck-live 5.4GB master migrated SMB->S3, job tracked in Jobs tab,
tags swapped smb->s3.
2026-06-05 03:37:59 +00:00
OpenCode
105d04729a fix(capture): explicit slice threading on VC-3 growing encoder
dnxhd with ffmpeg default frame-threading starts at ~0.27x realtime and
buffers a long pipeline before output flows, so the fc_pipe ring laps ~344
startup frames (spotty audio+video for the first several seconds). Setting
-threads 32 -thread_type slice makes dnxhd encode >= realtime from frame 1
(measured 1.3x at start), cutting startup drops substantially. The finalized
master file is gap-free (even PTS, audio==video duration) — the remaining
skipped frames are pre-record spin-up frames dropped to the live edge, not
holes in the recording.
2026-06-05 03:12:42 +00:00
OpenCode
35fb84af4d feat(capture): VC-3/DNxHD growing MXF for Premiere edit-while-record
Replace the AVC-Intra growing path (which Premiere rejected as unsupported/
damaged) with VC-3/DNxHD written directly by ffmpeg native MXF muxer. The
frame-wrapped OP1a body grows readably mid-write and imports+grows live in
Adobe Premiere (matches vMix workflow). No raw2bmx, no FIFO orchestrator, no
footer-finalize ordering - one ffmpeg writes MXF straight to the SMB share.

Two operator-selectable bitrates: vc3_90 (~90 Mbps, default) and vc3_220
(~220 Mbps). Both 8-bit 4:2:2 @ 1080p59.94, essence VC3_1080p_1238. Stop uses
a plain SIGINT (ffmpeg flushes the MXF footer cleanly).

UI: growing codec select (90/220) replaces the AVC-Intra readonly field; the
freeform growing bitrate input is removed (bitrate is codec-fixed). mam-api
guards accept vc3_90/vc3_220, default vc3_90.

Verified on zampp3: both bitrates grow live + finalize clean (check-complete
passes, 0 decode errors), user-confirmed Premiere import + growth.
2026-06-05 03:04:45 +00:00
OpenCode
f1f4f50714 chore: untrack .env.worker, add to gitignore 2026-06-04 22:58:00 +00:00
OpenCode
1e206a55fa feat(ui): version badge, polling fixes, asset browser hygiene, project ctx fixes 2026-06-04 22:54:49 +00:00
OpenCode
2a43deb0be chore: remove one-shot avci patch scripts 2026-06-04 22:10:22 +00:00
OpenCode
ec58556c36 fix(cluster): trust non-bridge ip_address from heartbeat payload 2026-06-04 22:09:15 +00:00
Zac Gaetano
6c1edae7f0 chore: add zampp3 compose override (3x L4, no DeckLink) 2026-06-04 22:08:24 +00:00
OpenCode
c8cddc19b2 feat: AVC-Intra 50/100/200 growing codec selector 2026-06-04 21:51:45 +00:00
071e1f6461 add UI patch script for AVC-Intra 50/100/200 growing codec options 2026-06-04 17:51:24 -04:00
42a591cc97 add patch script for call site + recorders.js growing_codec guard 2026-06-04 17:50:41 -04:00
a5573a918d add patch script for AVC-Intra 50/100/200 growing codec classes 2026-06-04 17:50:05 -04:00
ZGaetano
f7447e6ec2 feat(ui): show + select growing codec (AVCI-100 CPU vs HEVC-NVENC GPU) 2026-06-04 21:13:35 +00:00
ZGaetano
6af5d77d62 feat(growing): thread growing_codec through /capture/start session params 2026-06-04 20:58:20 +00:00
ZGaetano
a031ff1c9e feat(growing): add GPU-offload HEVC-NVENC frag-MOV growing codec
Second selectable growing path alongside AVC-Intra 100. GROWING_CODEC env
(per-recorder growing_codec field) picks: avci100 (CPU 4:2:2 MXF, default) or
hevc_nvenc (GPU all-intra HEVC 10-bit 4:2:0 fragmented MOV). The hevc path runs
a single ffmpeg (no raw2bmx/FIFO) writing frag-MOV directly; +empty_moov makes
it grow with readable duration mid-write. Proven live on zampp3: size+duration
advance monotonically, finalized file decodes RC=0 (hevc Main10 1080p59.94).
2026-06-04 20:53:13 +00:00
ZGaetano
967547ae97 fix(growing): drop FIFO FD-priming deadlock — ffmpeg-first ordering feeds raw2bmx frames
Root cause of frozen 22KB growing MXF: the parent held a stray read-write
priming writer (FD 7) on the video FIFO while raw2bmx opened it, so raw2bmx
read a malformed/empty stream start and exited after the header (Duration:0).
Proven live on zampp3: removing the FD-7/8 priming + fd-watch loop and simply
starting ffmpeg before raw2bmx lets the blocking FIFO opens pair up naturally
(ffmpeg sole writer). True-1080p59.94 AVC-Intra 100 then grows monotonically
on disk and a mid-write snapshot decodes to its last frame (481 frames @ 8s).
2026-06-04 19:31:48 +00:00
bf486b93dd feat(growing): TRUE 1080p59.94 via AVC-Intra 100 (libx264 4:2:2 10-bit)
User wants native 59.94p, not 30p. XDCAM HD422/MPEG-2 422 cannot do 1080p59.94 (raw2bmx rejects 60000/1001). AVC-Intra 100 CAN. Switched growing essence to libx264 high422 10-bit avcintra-class=100 (NVENC h264 cannot do 4:2:2, so CPU), -f h264, raw2bmx -t op1a --avci100_1080p -f 60000/1001. Verified via the live-equivalent fc_pipe|bash path WITH the FD-9 stdin fix: produces h264 High 4:2:2 Intra yuv422p10le 1920x1080 progressive 60000/1001 MXF (true 59.94p), openable in Premiere.
2026-06-04 19:02:11 +00:00
a00a280689 fix(growing): ffmpeg reads video from saved FD 9 — fixes empty-output growing
ROOT CAUSE FOUND + verified. When growing video comes from fc_pipe, node pipes it to the bash orchestrator's stdin. ffmpeg ran as a backgrounded subshell merely inheriting fd 0 (0<&0). With a PIPE source (not the working file/FIFO case), that subshell was starved of the raw video -> filtergraph 'No filtered frames' -> empty mpeg2video -> raw2bmx broken pipe -> sidecar crash (write EPIPE). Reproduced exactly with 'fc_pipe | bash -c orchestrator'. Fix: save original stdin to FD 9 BEFORE the FIFO-priming fd games (exec 9<&0), point ffmpeg's fd0 at fd9 (0<&9), close 9 in raw2bmx + parent. Verified the live-equivalent path now produces a valid mpeg2video 4:2:2 yuv422p progressive 30000/1001 MXF matching the working Delta7 file. Also added EPIPE handlers so a broken pipe never crashes the sidecar.
2026-06-04 18:59:53 +00:00
690f27218d fix(growing): parent shell drops fd0 so ffmpeg is sole stdin reader
Root cause of growing 'video empty / No filtered frames': when video comes from fc_pipe, node pipes it to the bash orchestrator's stdin (fd0). The ffmpeg subshell inherits fd0, BUT the parent bash kept fd0 open too (in its wait loop). Both held the read end of the same pipe, so the kernel split the raw video bytes between them and ffmpeg was starved -> zero decoded frames -> empty mpeg2video -> raw2bmx broken pipe. Manual 'fc_pipe | ffmpeg' worked because ffmpeg was the sole reader. Fix: parent execs 0</dev/null right after spawning ffmpeg, making ffmpeg the sole stdin reader. Verified the full pipeline (fc_pipe->ffmpeg mpeg2video 422->raw2bmx rdd9) produces a valid 1080p29.97 MXF matching the working Delta7 file.
2026-06-04 18:52:07 +00:00
c2e3ee7dd2 fix(growing): XDCAM HD422 1080p29.97 — exact match to working Delta7 file
ffprobe of the proven-working Delta7_20260603 file (Premiere opened it): mpeg2video 4:2:2 yuv422p 1920x1080 PROGRESSIVE 30000/1001 + 2x pcm_s16le. The 1080p59.94 source is frame-rate-halved to 1080p29.97 progressive (NOT interlaced, NOT 59.94p). Verified on-node: mpeg2video yuv422p -r 30000/1001 -> raw2bmx -t rdd9 --mpeg2lg_422p_hl_1080p -f 30000/1001 produces an MXF byte-identical in format to the working file (rc=0).
2026-06-04 18:39:29 +00:00
4609517756 fix(growing): AVC-Intra 100 1080p59.94 via libx264 (verified raw2bmx accepts)
Verified on-node: libx264 high422 10-bit + x264 avcintra-class=100 -> raw2bmx -t op1a --avci100_1080p -f 60000/1001 produces a valid MXF (ffprobe: h264 High 4:2:2 Intra yuv422p10le 60000/1001). True 1080p59.94, openable in Premiere. CPU encode for now per demo deadline; NVENC h264 cannot do 4:2:2.
2026-06-04 18:35:42 +00:00
e6b7856bb2 fix(growing): restore proven XDCAM HD422 (MPEG-2 422) rdd9 @1080i59.94
Today's codec churn (h264_nvenc/AVC-Intra/op1a) produced .mxf files Premiere could not open. Subagent diagnosis vs the old working files on the share proved the working format is XDCAM HD422 = mpeg2video 4:2:2 yuv422p, clip type rdd9, wrapped at 1080i59.94 (30000/1001). raw2bmx REJECTS MPEG-2 422 at 60000/1001, so a 1080p59.94 SDI feed must wrap as 1080i59.94. Reverted GROWING_VIDEO_ELEMENTARY_ARGS to mpeg2video, raw2bmx -t rdd9 --mpeg2lg_422p_hl_1080i -f 30000/1001, -f mpeg2video pipe, and rates() 59.94->30000/1001 with 1080 forced interlaced.
2026-06-04 18:31:03 +00:00
e6eb565e30 fix(growing): switch to NVENC H.264 High Intra
Using libx264 confirmed raw2bmx works with AVC-Intra parameters at 59.94. Switching back to hardware-accelerated h264_nvenc with matching parameters (High profile, all-intra GOP 1, yuv422p, aud) to keep CPU load low during 8-port burn tests.
2026-06-04 18:18:41 +00:00
4bbbc9f4dc fix(growing): add -aud 1 to h264_nvenc to fix raw2bmx parse failure 2026-06-04 18:12:50 +00:00
42806b5e10 fix(growing): use stable H.264 High All-Intra for 1080p59.94 MXF
XDCAM HD422 does not strictly support 1080p59.94, and ffmpeg/raw2bmx failed to negotiate the stream. Reverted to h264_nvenc (High profile, all-intra GOP 1, yuv420p) which raw2bmx can reliably wrap as OP1a (--avc_high) at 60000/1001. This restores NVENC hardware acceleration and Premiere edit-while-record compatibility.
2026-06-04 17:56:08 +00:00
f36de429e8 feat(growing): configure AVC-Intra 100 1080p59.94 via NVENC for growing files 2026-06-04 17:49:21 +00:00
1f71de494f fix(growing): correct growing frame rate selection for 59.94 fps 2026-06-04 17:31:43 +00:00
c2d15b4e3a debug: capture raw2bmx output to log 2026-06-04 17:11:47 +00:00
3c7cc1a77f fix(worker): retry transient S3 aborts + reuse one keep-alive client
Burn test: 5 assets errored during proxy with 'aborted'/'socket hang up'
during the master DOWNLOAD. The masters all exist in S3 (262-269MB) — it's
the connection-limited RustFS backend dropping streams when 8 jobs hammer it
at once. Two fixes:

1. downloadFromS3/uploadToS3 now retry transient failures (aborted, socket
   hang up, ECONNRESET, timeout, 5xx, throttle) up to 5x with exponential
   backoff, cleaning the partial file between download attempts. A single
   mid-stream abort no longer errors the whole asset.

2. Reuse ONE shared S3 client instead of createS3Client()+client.destroy()
   per call. The per-call destroy tore down the keep-alive agent's sockets
   every time, so connection pooling never happened and each transfer opened
   fresh connections — exactly what overwhelmed RustFS. A long-lived client
   lets the keep-alive pool actually be reused.
2026-06-04 16:56:11 +00:00
80f157968f fix(capture): pin NVENC to a GPU with ffmpeg -gpu N (privileged bypasses env)
NVIDIA_VISIBLE_DEVICES=1 was set but the sidecar still SAW /dev/nvidia0,1,2 and nvenc used GPU 0 — because capture sidecars run Privileged, which exposes every GPU device node regardless of NVIDIA_VISIBLE_DEVICES/DeviceRequests. Real fix: node-agent passes CAPTURE_GPU_INDEX to the sidecar and capture-manager adds ffmpeg '-gpu N' to the hevc_nvenc + h264_nvenc encoders, so each port's master+HLS encode is explicitly bound to its assigned L4. Spreads 8 ports across 3 cards.
2026-06-04 16:07:59 +00:00
15fab99d55 fix(node-agent): scope DeviceRequests to chosen GPU so VISIBLE_DEVICES sticks
NVIDIA_VISIBLE_DEVICES was set per-port correctly (0/1/2) but ALL encodes still landed on GPU 0 at 99%. Cause: HostConfig.DeviceRequests granted Count:-1 (all GPUs), which OVERRIDES NVIDIA_VISIBLE_DEVICES — the container saw all 3 cards and nvenc defaulted to device 0. Now build DeviceRequests with DeviceIDs:[chosenGpu] so each sidecar truly sees only its one L4.
2026-06-04 16:05:14 +00:00
f223bb8c8b fix(node-agent): count GPUs via /dev/nvidiaN device nodes
_gpuCache was empty (probeGpusViaSmi container didn't populate it), so the count fell back to 1 → NVIDIA_VISIBLE_DEVICES=all again. Count /dev/nvidiaN nodes directly (visible in the privileged node-agent container, confirmed 3) — same method the heartbeat uses.
2026-06-04 16:02:22 +00:00
15c282e749 fix(node-agent): GPU count from _gpuCache, not direct nvidia-smi
The node-agent image has no nvidia-smi binary, so the direct execFileSync
detect always failed → fell back to 1 GPU → NVIDIA_VISIBLE_DEVICES=all (the
exact bug we were fixing). Use the existing _gpuCache (populated at startup
by probeGpusViaSmi via a throwaway GPU container) for the count instead.
2026-06-04 15:59:48 +00:00
4be12c6f9a fix(stability): spread capture encodes across all GPUs + GOP parse + filmstrip retry
VIDEO FREEZE UNDER BURN (transient stall, self-recovers): all 8 capture
sidecars ran NVIDIA_VISIBLE_DEVICES=all with no -gpu selector, so ffmpeg
nvenc put every session (8 HEVC masters + 8 HLS = 16) on physical GPU 0
while the other two L4s sat idle. GPU 0 NVENC hit 86%, encode fell below
realtime, the framecache ring lapped → video froze → caught up → recovered.
Bridge verified smooth at 60fps throughout. FIX: node-agent now round-robins
each sidecar to a GPU by capture port (port % detected-GPU-count) via
NVIDIA_VISIBLE_DEVICES, honoring an explicit gpuUuid when set. Auto-detects
GPU count from nvidia-smi (override CAPTURE_GPU_COUNT). ~3 encoders/GPU now.

GOP PARSE: Number.parseFloat('60000/1001') returns 60000, making GOP 120000
(near open-GOP) instead of ~120. Added parseFps() to handle rational rates;
fixed hevcNvencArgs + buildHlsVideoArgs.

FILMSTRIP: RustFS object store intermittently returns NoSuchKey on GET for
keys that List/Head confirm exist, blanking the strip. Generation/queue/DB
all verified healthy (13/15 assets HAVE filmstrips). FIX: API now serves the
filmstrip JSON through itself with retry-on-NoSuchKey (succeeds within a
couple attempts) instead of handing the browser a signed URL — also closes
the S3 CORS gap. Frontend updated to consume the direct JSON.
2026-06-04 15:56:44 +00:00