diff --git a/docker-compose.worker.yml b/docker-compose.worker.yml index 629516e..110ccd0 100644 --- a/docker-compose.worker.yml +++ b/docker-compose.worker.yml @@ -103,6 +103,7 @@ services: # worker-l4: HEAVY tier (proxy/conform/trim) on the L4 (NVENC). Talks to # zampp1's Redis/Postgres/S3 over the LAN (.200). No promotion scanner here. worker-l4: + profiles: [gpu] build: context: ./services/worker dockerfile: Dockerfile.gpu diff --git a/docs/superpowers/specs/2026-05-31-storage-settings-growing-smb-design.md b/docs/superpowers/specs/2026-05-31-storage-settings-growing-smb-design.md new file mode 100644 index 0000000..863aa9c --- /dev/null +++ b/docs/superpowers/specs/2026-05-31-storage-settings-growing-smb-design.md @@ -0,0 +1,148 @@ +# Storage Settings Warning + Growing-Files SMB Auth + Per-Recorder Growing Mode +**Date:** 2026-05-31 +**Branch:** `feat/playout-mcr` (Forgejo: WildDragonLLC/dragonflight) +**Status:** Approved, ready for implementation plan + +--- + +## Scope + +Three related refinements to the Settings page and growing-files capture pipeline: + +1. **Storage warning header** — a prominent set-once warning at the top of the Storage settings section. +2. **Growing-files SMB credentials + system CIFS mount** — store an SMB username/password and have the capture stack mount the growing share itself (Approach A). +3. **Per-recorder growing mode** — remove the global "capture writes to local SMB share first" toggle; make growing-files mode a per-recorder setting. + +All changes live on the `growing-files` / storage-settings path. No playout changes (handled separately). + +--- + +## Background (current state) + +- **Settings storage:** single key/value `settings(key TEXT PRIMARY KEY, value TEXT, updated_at)` table (migration 006). Secrets like `s3_secret_key` are stored but `GET /settings/s3` returns only `s3_secret_key_exists` (never the value). +- **Growing settings UI:** `GrowingSettingsCard` in `services/web-ui/public/screens-admin.jsx` (rendered by `StorageSection` alongside `MountHealthStrip` and `S3SettingsCard`). Current keys: `growing_enabled`, `growing_path`, `growing_smb_url`, `growing_promote_after_seconds`. +- **Settings API:** `services/mam-api/src/routes/settings.js` — `GET/PUT /settings/growing` over `GROWING_KEYS`. +- **Global enable today:** the `growing_enabled` setting (checkbox labelled "Capture writes to the local SMB share first; Premier can edit while it's still growing"). `recorders.js` reads this global key at recorder-start and passes `GROWING_ENABLED=true/false` to the capture container. +- **Capture write path:** `services/capture/src/capture-manager.js` reads `process.env.GROWING_ENABLED` and `GROWING_PATH` (default `/growing`). When on, it writes the master to `/growing/{projectId}/{clipName}.{ext}` instead of streaming to S3; the promotion worker uploads to S3 after stop. +- **Current mount model:** `/growing` is a pre-mounted host bind-mount; the app never authenticates to SMB. +- **Per-recorder column already exists:** migration 014 added `recorders.growing_enabled BOOLEAN DEFAULT NULL` ("NULL = use global"), but recorder-start logic ignores it and the new-recorder modal does not expose it. + +--- + +## Part 1 — Storage warning header + +Add a danger-styled banner at the **top of `StorageSection()`**, above `MountHealthStrip`. + +- Visual: full-width banner, danger token styling (`--danger` border + subtle danger background), alert icon, bold uppercase text. +- Exact copy: + > **⚠ WARNING — THESE SETTINGS ARE MEANT TO BE SET ONCE AT INITIAL DEPLOYMENT. CHANGING STORAGE PATHS MID-CYCLE CAN CAUSE DATABASE CORRUPTION AND FILE LOSS. PLEASE USE WITH CAUTION.** +- Pure presentational; no backend, no dismiss state (always visible). + +**Files:** `services/web-ui/public/screens-admin.jsx` (and a small style rule in the appropriate CSS file if needed). + +--- + +## Part 2 — Growing-files SMB credentials + system CIFS mount (Approach A) + +The growing share is **shared infrastructure**, so the SMB connection config is global. + +### New settings keys +| Key | Purpose | Notes | +|-----|---------|-------| +| `growing_smb_mount` | CIFS source for the system mount, e.g. `//10.0.0.25/mam-growing` | Distinct from `growing_smb_url` | +| `growing_smb_username` | SMB user | Returned in GET (not secret) | +| `growing_smb_password` | SMB password | **Write-only** — never returned | +| `growing_smb_vers` *(optional)* | CIFS protocol version, default `3.0` | Avoids mount negotiation failures | + +`growing_smb_url` (the `smb://…` string) is retained unchanged as the **editor-facing** display value (Premiere connect string). + +### Settings API (`settings.js`) +- Extend `GROWING_KEYS` with the new keys (except the password is handled specially). +- `GET /settings/growing`: return `growing_smb_mount`, `growing_smb_username`, `growing_smb_vers`, and `growing_smb_password_exists: boolean` — **never** the password value. (Mirror the existing `s3_secret_key_exists` pattern.) +- `PUT /settings/growing`: upsert each provided key. For `growing_smb_password`, only write it when a non-empty value is provided (an empty/omitted field leaves the stored password unchanged). Provide a way to clear it (explicit empty sentinel or a "clear" affordance) — see Resolved decisions below. + +### Settings UI (`GrowingSettingsCard`) +Add three fields to the card: +- **SMB mount (CIFS):** text input bound to `growing_smb_mount`, placeholder `//10.0.0.25/mam-growing`. +- **SMB username:** text input bound to `growing_smb_username`. +- **SMB password:** masked password input. Shows a "saved" indicator when `growing_smb_password_exists` is true; typing a new value replaces it; leaving it blank keeps the existing one. `autoComplete="new-password"`. + +### Capture image (`services/capture/Dockerfile`) +Add `cifs-utils` to the installed packages so `mount -t cifs` is available inside the capture container. + +### Capture-manager (`capture-manager.js`) +On capture start, when growing mode is active **and** `GROWING_SMB_MOUNT` is set: +1. Write a root-only credentials file (e.g. `/run/smb-creds`, mode `0600`) containing: + ``` + username= + password= + ``` + (Credentials go in the file, **not** the mount command line, to avoid `ps`/log exposure.) +2. `mkdir -p $GROWING_PATH` then `mount -t cifs $GROWING_SMB_MOUNT $GROWING_PATH -o credentials=/run/smb-creds,uid=0,gid=0,file_mode=0664,dir_mode=0775,vers=$GROWING_SMB_VERS`. +3. If the mount succeeds → proceed writing the master to `$GROWING_PATH/...` (existing behaviour). +4. If the mount **fails** → log the error and fall back to S3 streaming (`growingPath = null`), so a recording is never lost. +5. On capture stop/cleanup, unmount `$GROWING_PATH` (best-effort; ignore "not mounted"). + +Mount isolation: each recorder runs in its **own** capture container, so each container mounts CIFS at its own private `/growing` — no cross-recorder mount conflicts, no ref-counting needed. + +### Recorder start (`recorders.js`) +- Pass new env to the spawned capture container: `GROWING_SMB_MOUNT`, `GROWING_SMB_USERNAME`, `GROWING_SMB_PASSWORD`, `GROWING_SMB_VERS` (read from the `settings` table at start). +- The dynamically-spawned capture container must get `/growing` as an **empty mountpoint** (not a host bind-mount) so the in-container CIFS mount lands cleanly. Confirm/adjust the container spec accordingly. The container is already privileged (required for `mount`). + +### Security notes +- The password is stored plaintext in the `settings` table, identical to the existing `s3_secret_key` handling — acceptable within this app's current secret model. +- The password reaches the capture container as an env var (visible via `docker inspect`), same as S3 keys already are. The credentials **file** (not the command line) is used for the actual mount. + +--- + +## Part 3 — Per-recorder growing mode (remove the global toggle) + +### Remove global enable +- Delete the global "Capture writes to the local SMB share first…" checkbox (`growing_enabled` key) from `GrowingSettingsCard`. The card no longer carries a global on/off — it is **infrastructure-only**: container mount path, SMB URL (editor), SMB mount + credentials, promote threshold. +- The `growing_enabled` settings *key* is retired from the UI. (It may remain in the table harmlessly; recorder-start no longer reads it.) + +### Per-recorder semantics +- Reuse the existing `recorders.growing_enabled BOOLEAN` column. New semantics (no global to defer to): `TRUE` = this recorder uses growing-files mode; `NULL`/`FALSE` = off. +- `recorders.js` recorder-start: read the **recorder's own** `growing_enabled` (defaulting `NULL`→off) and set `GROWING_ENABLED` for the capture container from that, instead of the global setting. +- Add `growing_enabled` to `RECORDER_FIELDS` so create/update accept it. + +### UI +- **New-recorder modal** (`modal-new-recorder.jsx`): add a "Growing-files mode" toggle that sets `growing_enabled` on the created recorder (default off). +- **Recorder edit** (wherever recorders are edited): same toggle. +- Helper text on the toggle notes that growing-files requires the SMB share to be configured in Settings → Storage. + +### Fallback +If a recorder has `growing_enabled = true` but `growing_smb_mount` is not configured globally, capture logs a warning and falls back to S3 streaming (same fallback path as a failed mount). Recording is never blocked. + +--- + +## Files changed + +| File | Change | +|------|--------| +| `services/web-ui/public/screens-admin.jsx` | Storage warning banner; SMB mount/username/password fields in `GrowingSettingsCard`; remove global growing-enable checkbox | +| `services/web-ui/public/modal-new-recorder.jsx` | Per-recorder "Growing-files mode" toggle | +| `services/mam-api/src/routes/settings.js` | New growing SMB keys; write-only password (`growing_smb_password_exists`) | +| `services/mam-api/src/routes/recorders.js` | Read per-recorder `growing_enabled`; pass SMB env to capture; `RECORDER_FIELDS` += `growing_enabled`; empty `/growing` mountpoint | +| `services/capture/Dockerfile` | Add `cifs-utils` | +| `services/capture/src/capture-manager.js` | CIFS mount-on-start (creds file), unmount-on-stop, fallback to S3 on failure | +| CSS (storage warning / fields) | Minor styles if needed | + +No DB migration required (the `recorders.growing_enabled` column already exists; new settings are key/value rows). + +--- + +## Resolved decisions + +- **Clearing the SMB password:** `PUT /settings/growing` treats a field value of the literal sentinel `""` with an explicit `growing_smb_password_clear: true` flag as "remove the stored password"; a blank field with no clear flag leaves it unchanged. (Keeps the common "don't retype the password on every save" UX while still allowing removal.) +- **CIFS version:** default `growing_smb_vers = 3.0`; overridable via settings to support older NAS targets. +- **Recorders already recording when the toggle changes:** the per-recorder `growing_enabled` is read at **start** only; changing it mid-recording has no effect on the active session (consistent with how all recorder encode settings already behave). + +--- + +## Out of scope (deferred) + +- Encrypting secrets at rest (the app's existing model stores `s3_secret_key` in plaintext; SMB password follows the same model). +- A global "growing-files master kill switch" (removed by design — control is now per-recorder). +- Exposing `growing_retention_days` in the UI (seeded in DB, still unsurfaced; unrelated to this work). +- Playout HLS preview fix (handled by a separate parallel effort). diff --git a/services/capture/Dockerfile b/services/capture/Dockerfile index 1f94e4e..455dc2c 100644 --- a/services/capture/Dockerfile +++ b/services/capture/Dockerfile @@ -67,10 +67,14 @@ RUN /usr/local/bin/ffmpeg -hide_banner -encoders 2>&1 | grep -E 'nvenc' \ # ── Stage 2: Runtime image ─────────────────────────────────────────────────── FROM node:20-bookworm -# Runtime deps for compiled ffmpeg libs +# Runtime deps for compiled ffmpeg libs. +# cifs-utils provides mount.cifs so growing-files capture can mount the SMB +# landing-zone share inside the (privileged) container at start (Approach A). +# util-linux supplies mount/umount/mountpoint. RUN apt-get update && apt-get install -y --no-install-recommends \ libx264-164 libx265-199 libvpx7 libopus0 libmp3lame0 \ libsrt1.5-openssl libzmq5 libstdc++6 libc++1 libc++abi1 \ + cifs-utils util-linux \ && rm -rf /var/lib/apt/lists/* # Copy compiled ffmpeg/ffprobe diff --git a/services/capture/src/capture-manager.js b/services/capture/src/capture-manager.js index 5f1315f..50108d8 100644 --- a/services/capture/src/capture-manager.js +++ b/services/capture/src/capture-manager.js @@ -1,5 +1,5 @@ -import { spawn } from 'child_process'; -import { mkdirSync } from 'node:fs'; +import { spawn, execFileSync } from 'child_process'; +import { mkdirSync, writeFileSync } from 'node:fs'; import { dirname } from 'node:path'; import { v4 as uuidv4 } from 'uuid'; import { createUploadStream } from './s3/client.js'; @@ -9,11 +9,76 @@ const S3_BUCKET = process.env.S3_BUCKET || 'wild-dragon'; // Growing-files mode: writes the master to a local SMB-backed share that the // editor can mount, instead of streaming to S3 in real time. The promotion // worker uploads the finalized file to S3 after the recording stops. -// Toggled per-process by `GROWING_ENABLED=true` on the capture container +// Toggled per-recorder via `GROWING_ENABLED=true` on the capture container // (see routes/recorders.js where the env is composed). const GROWING_ENABLED = process.env.GROWING_ENABLED === 'true'; const GROWING_PATH = process.env.GROWING_PATH || '/growing'; +// Approach A: when a CIFS source is supplied, this (privileged) container mounts +// the SMB landing-zone share at GROWING_PATH itself, using credentials supplied +// by the API from global Settings. Empty GROWING_SMB_MOUNT → no system mount +// (the host-bound /growing volume is used instead, or S3 streaming if growing +// is off). +const GROWING_SMB_MOUNT = process.env.GROWING_SMB_MOUNT || ''; +const GROWING_SMB_USERNAME = process.env.GROWING_SMB_USERNAME || ''; +const GROWING_SMB_PASSWORD = process.env.GROWING_SMB_PASSWORD || ''; +const GROWING_SMB_VERS = process.env.GROWING_SMB_VERS || '3.0'; +const SMB_CREDS_FILE = '/run/smb-creds'; + +// True when GROWING_PATH is already a mountpoint (e.g. a prior session left it +// mounted, or a host bind-mount is present). +function isMounted(path) { + try { execFileSync('mountpoint', ['-q', path]); return true; } + catch { return false; } +} + +// Mount the CIFS growing share at GROWING_PATH. Credentials go in a root-only +// file (NOT the command line) so they never appear in `ps`/process listings. +// Returns true on success (or if already mounted), false on failure — callers +// fall back to S3 streaming so a recording is never lost. +function mountGrowingShare() { + if (!GROWING_SMB_MOUNT) return false; + try { + if (isMounted(GROWING_PATH)) { + console.log('[capture] growing share already mounted at', GROWING_PATH); + return true; + } + try { mkdirSync(GROWING_PATH, { recursive: true }); } catch (_) {} + writeFileSync( + SMB_CREDS_FILE, + `username=${GROWING_SMB_USERNAME}\npassword=${GROWING_SMB_PASSWORD}\n`, + { mode: 0o600 } + ); + const opts = [ + `credentials=${SMB_CREDS_FILE}`, + 'uid=0', 'gid=0', 'file_mode=0664', 'dir_mode=0775', + `vers=${GROWING_SMB_VERS}`, + ].join(','); + execFileSync('mount', ['-t', 'cifs', GROWING_SMB_MOUNT, GROWING_PATH, '-o', opts], + { stdio: ['ignore', 'ignore', 'pipe'] }); + console.log('[capture] mounted CIFS growing share', GROWING_SMB_MOUNT, '->', GROWING_PATH); + return true; + } catch (err) { + const stderr = err.stderr ? err.stderr.toString().trim() : err.message; + console.error('[capture] CIFS mount failed (falling back to S3 streaming):', stderr); + return false; + } +} + +// Best-effort unmount on session stop. Ignores "not mounted". +function unmountGrowingShare() { + if (!GROWING_SMB_MOUNT) return; + try { + if (isMounted(GROWING_PATH)) { + execFileSync('umount', [GROWING_PATH], { stdio: ['ignore', 'ignore', 'pipe'] }); + console.log('[capture] unmounted growing share at', GROWING_PATH); + } + } catch (err) { + const stderr = err.stderr ? err.stderr.toString().trim() : err.message; + console.warn('[capture] growing share unmount failed (ignored):', stderr); + } +} + // ── Codec catalogue ────────────────────────────────────────────────────── // Each entry maps the UI value to a base ffmpeg flag set. Bitrate / framerate // / pix_fmt are layered on top from the per-recorder configuration. @@ -283,7 +348,15 @@ class CaptureManager { // Growing-files: write master to the local SMB share instead of streaming // to S3. Path is relative to the container's GROWING_PATH mount. - const growingPath = GROWING_ENABLED + // + // Approach A: if a CIFS source is configured, mount it now. A mount failure + // is non-fatal — we fall back to S3 streaming so the recording is never + // lost. + let growingActive = GROWING_ENABLED; + if (growingActive && GROWING_SMB_MOUNT) { + if (!mountGrowingShare()) growingActive = false; // fall back to S3 + } + const growingPath = growingActive ? `${GROWING_PATH}/${projectId}/${clipName}.${hiresExt}` : null; if (growingPath) { @@ -455,6 +528,11 @@ class CaptureManager { if (processes.proxy) processes.proxy.kill('SIGINT'); if (processes.hls) { try { processes.hls.kill('SIGINT'); } catch (_) {} } + // Release the CIFS mount (best-effort) once the ffmpeg writers are done with + // it. The promotion worker reads the staged file from the host/S3 side, not + // through this container's mount, so unmounting here is safe. + unmountGrowingShare(); + try { const uploadPromises = [currentSession.uploads.hires]; if (currentSession.uploads.proxy) uploadPromises.push(currentSession.uploads.proxy); diff --git a/services/mam-api/src/routes/assets.js b/services/mam-api/src/routes/assets.js index 038df5c..8e3463d 100644 --- a/services/mam-api/src/routes/assets.js +++ b/services/mam-api/src/routes/assets.js @@ -734,11 +734,14 @@ router.get('/:id/live-path', async (req, res, next) => { if (a.rows.length === 0) return res.status(404).json({ error: 'Asset not found' }); const asset = a.rows[0]; if (asset.status !== 'live') return res.status(409).json({ error: 'Asset is not currently growing', status: asset.status }); - const s = await pool.query(`SELECT key, value FROM settings WHERE key IN ('growing_enabled','growing_smb_url')`); + // Growing-files mode is now per-recorder (recorders.growing_enabled), so we + // no longer gate on the removed global `growing_enabled` setting. A + // status='live' asset already proves a growing recorder is producing this + // file; we only need the editor-facing SMB URL to build the UNC path. + const s = await pool.query(`SELECT key, value FROM settings WHERE key = 'growing_smb_url'`); const cfg = {}; for (const { key, value } of s.rows) cfg[key] = value; - if (cfg.growing_enabled !== 'true') return res.status(409).json({ error: 'Growing-files mode is disabled' }); - if (!cfg.growing_smb_url) return res.status(409).json({ error: 'No SMB URL configured — set growing_smb_url in Settings' }); + if (!cfg.growing_smb_url) return res.status(409).json({ error: 'No SMB URL configured — set the editor SMB URL in Settings → Storage' }); const rec = await pool.query( `SELECT recording_container FROM recorders WHERE current_session_id = $1 ORDER BY updated_at DESC LIMIT 1`, [asset.id] diff --git a/services/mam-api/src/routes/cluster.js b/services/mam-api/src/routes/cluster.js index d1cb842..6fe3dde 100644 --- a/services/mam-api/src/routes/cluster.js +++ b/services/mam-api/src/routes/cluster.js @@ -1,9 +1,27 @@ import express from 'express'; import http from 'http'; import pool from '../db/pool.js'; +import { requireAdmin } from '../middleware/auth.js'; const router = express.Router(); +// GET /onboard-info – admin-only. Supplies the Add Node wizard with the bits it +// needs to build a `curl … | bash` onboarding command: the primary API URL the +// remote node-agent should heartbeat to, the raw URL of onboard-node.sh, and +// the deploy branch. apiUrl is a best guess the UI lets the operator edit. +router.get('/onboard-info', requireAdmin, (req, res) => { + const branch = process.env.DEPLOY_BRANCH || 'main'; + const apiUrl = process.env.PUBLIC_API_URL + || `${req.protocol}://${req.hostname}:${process.env.API_PORT || 47432}`; + const scriptUrl = + `https://forge.wilddragon.net/zgaetano/wild-dragon/raw/branch/${branch}/deploy/onboard-node.sh`; + res.json({ apiUrl, scriptUrl, branch }); +}); + +// If the agent reported Docker's default bridge IP (172.17.x) but the request +// itself came from a real LAN address, prefer the request source IP instead. +// We only check 172.17.x — the default docker0 bridge — not the full RFC1918 +// 172.16/12 block, since real LANs (e.g. 172.18.91.x) fall in that range. function pickIp(reportedIp, reqIp) { const clean = (s) => (s || '').replace(/^::ffff:/, ''); const isDockerBridge = (ip) => /^172\.17\./.test(ip || ''); diff --git a/services/mam-api/src/routes/jobs.js b/services/mam-api/src/routes/jobs.js index 9efa751..5608cb1 100644 --- a/services/mam-api/src/routes/jobs.js +++ b/services/mam-api/src/routes/jobs.js @@ -22,20 +22,22 @@ const parseRedisUrl = (url) => { const redisConn = parseRedisUrl(process.env.REDIS_URL || 'redis://queue:6379'); -const proxyQueue = new Queue('proxy', { connection: redisConn }); -const thumbnailQueue = new Queue('thumbnail', { connection: redisConn }); -const filmstripQueue = new Queue('filmstrip', { connection: redisConn }); -const conformQueue = new Queue('conform', { connection: redisConn }); -const importQueue = new Queue('import', { connection: redisConn }); -const trimQueue = new Queue('trim', { connection: redisConn }); +const proxyQueue = new Queue('proxy', { connection: redisConn }); +const thumbnailQueue = new Queue('thumbnail', { connection: redisConn }); +const filmstripQueue = new Queue('filmstrip', { connection: redisConn }); +const conformQueue = new Queue('conform', { connection: redisConn }); +const importQueue = new Queue('import', { connection: redisConn }); +const trimQueue = new Queue('trim', { connection: redisConn }); +const playoutStageQueue = new Queue('playout-stage', { connection: redisConn }); const QUEUES = [ - { queue: proxyQueue, type: 'proxy' }, - { queue: thumbnailQueue, type: 'thumbnail' }, - { queue: filmstripQueue, type: 'filmstrip' }, - { queue: conformQueue, type: 'conform' }, - { queue: importQueue, type: 'import' }, - { queue: trimQueue, type: 'trim' }, + { queue: proxyQueue, type: 'proxy' }, + { queue: thumbnailQueue, type: 'thumbnail' }, + { queue: filmstripQueue, type: 'filmstrip' }, + { queue: conformQueue, type: 'conform' }, + { queue: importQueue, type: 'import' }, + { queue: trimQueue, type: 'trim' }, + { queue: playoutStageQueue, type: 'playout-stage' }, ]; // BullMQ state → API status mapping diff --git a/services/mam-api/src/routes/playout.js b/services/mam-api/src/routes/playout.js index de1dcbb..f002c9b 100644 --- a/services/mam-api/src/routes/playout.js +++ b/services/mam-api/src/routes/playout.js @@ -11,6 +11,7 @@ import express from 'express'; import http from 'http'; +import { readFile } from 'fs/promises'; import { Queue } from 'bullmq'; import pool from '../db/pool.js'; import { validateUuid } from '../middleware/errors.js'; @@ -307,9 +308,14 @@ async function spawnChannelSidecar(channel) { } } + // Set last_heartbeat_at = NOW() so the scheduler health tick treats this + // channel as freshly alive. Without this, last_heartbeat_at starts as NULL + // (epoch=0), and the very first tick sees ageMs >> TIMEOUT_MS and triggers + // failover immediately — before the sidecar has had a chance to respond. const { rows } = await pool.query( `UPDATE playout_channels - SET status = 'running', container_id = $1, container_meta = $2, updated_at = NOW() + SET status = 'running', container_id = $1, container_meta = $2, + last_heartbeat_at = NOW(), updated_at = NOW() WHERE id = $3 RETURNING *`, [containerId, JSON.stringify(containerMeta), channel.id] ); @@ -367,6 +373,39 @@ router.get('/channels/:id/status', async (req, res, next) => { } }); +// GET /playout/channels/:id/hls/index.m3u8 — the live preview playlist, served +// through the API (not the static /media/live path) so it bypasses the public +// reverse proxy's static cache. That proxy caches the .m3u8 by path with a +// multi-second TTL and ignores the origin's no-store, so hls.js's ~1s reloads +// always got a STALE playlist ("MISSED" forever → monitor stayed black). The +// /api/ path is not proxy-cached (the status poll updates fine), so this always +// returns the fresh live edge. Segment (.ts) lines are rewritten to absolute +// /media/live// URLs so they still load from the static path (immutable, +// caching them is fine). mam-api shares the same /media volume the sidecars +// write to. +const HLS_MEDIA_DIR = process.env.PLAYOUT_MEDIA_DIR || '/media'; +router.get('/channels/:id/hls/index.m3u8', async (req, res, next) => { + try { + const cid = req.channel.id; + let body; + try { + body = await readFile(`${HLS_MEDIA_DIR}/live/${cid}/index.m3u8`, 'utf8'); + } catch (e) { + return res.status(404).json({ error: 'No live preview for this channel yet' }); + } + // Rewrite bare segment names to absolute static URLs. + const rewritten = body + .split('\n') + .map((line) => (/^[^#].*\.ts\s*$/.test(line) ? `/media/live/${cid}/${line.trim()}` : line)) + .join('\n'); + res.setHeader('Content-Type', 'application/vnd.apple.mpegurl'); + res.setHeader('Cache-Control', 'no-store, no-cache, must-revalidate'); + res.setHeader('Pragma', 'no-cache'); + res.send(rewritten); + } catch (err) { next(err); } +}); + +// ── Transport ──────────────────────────────────────────────────────────────── async function transport(req, res, action, body = null) { if (req.channel.status !== 'running') { return res.status(409).json({ error: 'Channel is not running' }); @@ -411,7 +450,14 @@ router.post('/channels/:id/play', requireChannelEdit, async (req, res, next) => asset_duration_ms: i.asset_duration_ms != null ? Number(i.asset_duration_ms) : null, })), }; - const out = await callSidecar(req.channel, '/playlist/load', 'POST', payload); + // callSidecar throws on network/timeout errors. Return 502 (not 409) so + // the UI and operators know it's a gateway problem, not a state conflict. + let out; + try { + out = await callSidecar(req.channel, '/playlist/load', 'POST', payload); + } catch (err) { + return res.status(502).json({ error: 'Sidecar unreachable: ' + err.message }); + } res.json(out); } catch (err) { next(err); } }); diff --git a/services/mam-api/src/routes/recorders.js b/services/mam-api/src/routes/recorders.js index 8367500..017ac83 100644 --- a/services/mam-api/src/routes/recorders.js +++ b/services/mam-api/src/routes/recorders.js @@ -154,6 +154,7 @@ const RECORDER_FIELDS = [ 'proxy_audio_codec', 'proxy_audio_bitrate', 'proxy_audio_channels', 'proxy_container', 'project_id', 'node_id', 'device_index', + 'growing_enabled', ]; function pickRecorderFields(body) { @@ -363,14 +364,25 @@ router.post('/:id/start', requireRecorderEdit, async (req, res, next) => { const externalMamApiUrl = `http://${process.env.NODE_IP || '172.18.91.200'}:${process.env.PORT_MAM_API || 47432}`; const dockerNetwork = process.env.DOCKER_NETWORK || 'wild-dragon_wild-dragon'; - // Growing-files mode is a global setting (settings table). When on, the - // capture container writes the master to its /growing/ mount instead of - // streaming it to S3 — Premiere can mount the SMB share and edit it live. - const growingRow = await pool.query( - `SELECT value FROM settings WHERE key = 'growing_enabled'` - ); - const growingEnabled = - growingRow.rows[0]?.value === 'true' || growingRow.rows[0]?.value === true; + // Growing-files mode is a PER-RECORDER setting (recorders.growing_enabled). + // When on, the capture container writes the master to its /growing/ mount + // instead of streaming it to S3 — editors can mount the SMB share and cut it + // live. The SMB share itself (mount source + credentials) is shared + // infrastructure configured globally in Settings → Storage. + const growingEnabled = recorder.growing_enabled === true; + + // Shared growing-files SMB infrastructure (global settings). Used to mount + // the CIFS share inside the capture container (services/capture mounts it + // with these credentials when GROWING_SMB_MOUNT is set). + const growingInfra = {}; + { + const r = await pool.query( + `SELECT key, value FROM settings WHERE key = ANY($1)`, + [['growing_smb_mount', 'growing_smb_username', 'growing_smb_password', 'growing_smb_vers']] + ); + for (const { key, value } of r.rows) growingInfra[key] = value; + } + const smbMount = growingEnabled ? (growingInfra.growing_smb_mount || '') : ''; // Operator-supplied clip name wins over the auto-timestamped fallback. // The Recorders UI passes this on the start request when the user types @@ -455,6 +467,13 @@ router.post('/:id/start', requireRecorderEdit, async (req, res, next) => { `MAM_API_TOKEN=${process.env.CAPTURE_TOKEN || ''}`, `GROWING_ENABLED=${growingEnabled ? 'true' : 'false'}`, `GROWING_PATH=/growing`, + // SMB mount details for the in-container CIFS mount (Approach A). Empty + // GROWING_SMB_MOUNT → capture falls back to the host-bound /growing volume + // (or to S3 streaming if growing isn't enabled). + `GROWING_SMB_MOUNT=${smbMount}`, + `GROWING_SMB_USERNAME=${growingInfra.growing_smb_username || ''}`, + `GROWING_SMB_PASSWORD=${growingInfra.growing_smb_password || ''}`, + `GROWING_SMB_VERS=${growingInfra.growing_smb_vers || '3.0'}`, ]; // Deltacast: pass port count so the capture container can enumerate @@ -530,7 +549,15 @@ router.post('/:id/start', requireRecorderEdit, async (req, res, next) => { for (const d of dcEntries) hostBinds.push(`/dev/${d}:/dev/${d}`); } catch (_) { /* no /dev/deltacast* nodes on this host */ } } - if (growingEnabled) hostBinds.push('/mnt/NVME/MAM/wild-dragon-growing:/growing'); + // /growing handling: + // - SMB mount configured → DON'T host-bind; the capture container mounts + // the CIFS share at /growing itself (Approach A). A bind-mount here + // would shadow the in-container mount. + // - growing on but no SMB mount → legacy host bind-mount fallback. + // - growing off → no /growing mount at all. + if (growingEnabled && !smbMount) { + hostBinds.push('/mnt/NVME/MAM/wild-dragon-growing:/growing'); + } const localEnv = [...env]; if (useGpu) { diff --git a/services/mam-api/src/routes/settings.js b/services/mam-api/src/routes/settings.js index f7cefdc..82fbe17 100644 --- a/services/mam-api/src/routes/settings.js +++ b/services/mam-api/src/routes/settings.js @@ -258,21 +258,45 @@ router.put('/transcoding', async (req, res, next) => { // while it's still being written; the promotion worker later moves the // finalized file to S3 and flips the asset to status='ready'. -const GROWING_KEYS = ['growing_enabled', 'growing_path', 'growing_smb_url', 'growing_promote_after_seconds']; +// Growing-files mode is now a PER-RECORDER setting (recorders.growing_enabled); +// the legacy global `growing_enabled` key is no longer read at recorder start. +// These global keys describe the shared SMB landing-zone infrastructure only: +// - growing_path container mount point (default /growing) +// - growing_smb_url smb://... display string for editors (Premiere) +// - growing_smb_mount //host/share CIFS source the capture container mounts +// - growing_smb_username SMB user for the system-side CIFS mount +// - growing_smb_password SMB password (WRITE-ONLY; never returned) +// - growing_smb_vers CIFS protocol version (default 3.0) +// - growing_promote_after_seconds idle threshold before S3 promotion +const GROWING_KEYS = [ + 'growing_path', 'growing_smb_url', 'growing_smb_mount', + 'growing_smb_username', 'growing_smb_vers', 'growing_promote_after_seconds', +]; +// growing_smb_password is handled separately: stored on PUT but NEVER returned +// on GET (only a *_exists flag), mirroring s3_secret_key. router.get('/growing', async (req, res, next) => { try { const result = await pool.query( `SELECT key, value FROM settings WHERE key = ANY($1)`, - [GROWING_KEYS] + [[...GROWING_KEYS, 'growing_smb_password']] ); const out = { - growing_enabled: 'false', growing_path: '/growing', growing_smb_url: '', + growing_smb_mount: '', + growing_smb_username: '', + growing_smb_vers: '3.0', growing_promote_after_seconds: '8', + growing_smb_password_exists: false, }; - for (const { key, value } of result.rows) out[key] = value; + for (const { key, value } of result.rows) { + if (key === 'growing_smb_password') { + out.growing_smb_password_exists = !!(value && value.length); + } else { + out[key] = value; + } + } res.json(out); } catch (err) { next(err); @@ -290,6 +314,19 @@ router.put('/growing', async (req, res, next) => { ); } } + // SMB password is write-only. A non-empty value sets/replaces it. To remove + // it, send growing_smb_password_clear:true. A blank/omitted password field + // leaves the stored value untouched (so operators don't retype it on every + // save). + if (req.body.growing_smb_password_clear === true) { + await pool.query(`DELETE FROM settings WHERE key = 'growing_smb_password'`); + } else if (typeof req.body.growing_smb_password === 'string' && req.body.growing_smb_password.length > 0) { + await pool.query( + `INSERT INTO settings (key, value, updated_at) VALUES ('growing_smb_password', $1, NOW()) + ON CONFLICT (key) DO UPDATE SET value = $1, updated_at = NOW()`, + [req.body.growing_smb_password] + ); + } res.json({ message: 'Growing-files settings saved' }); } catch (err) { next(err); diff --git a/services/mam-api/src/routes/storage.js b/services/mam-api/src/routes/storage.js index 9d767a8..493d87b 100644 --- a/services/mam-api/src/routes/storage.js +++ b/services/mam-api/src/routes/storage.js @@ -14,10 +14,12 @@ const exec = promisify(execCb); const router = express.Router(); // Defaults mirrored from settings.js so the overview never returns nulls. +// Growing-file mode is now per-recorder; "enabled" here means the shared SMB +// landing zone is CONFIGURED (a mount source is set), not a global on/off. const GROWING_DEFAULTS = { - growing_enabled: 'false', growing_path: '/growing', growing_smb_url: '', + growing_smb_mount: '', growing_promote_after_seconds: '8', }; @@ -100,7 +102,9 @@ router.get('/overview', async (req, res, next) => { try { // Growing files — merge defaults with whatever's in `settings`. const growingRaw = { ...GROWING_DEFAULTS, ...(await readSettings(Object.keys(GROWING_DEFAULTS))) }; - const growingEnabled = growingRaw.growing_enabled === 'true' || growingRaw.growing_enabled === true; + // "enabled" now means the shared SMB landing zone is configured (a mount + // source is set). Per-recorder toggles decide which recorders actually use it. + const growingEnabled = !!(growingRaw.growing_smb_mount && growingRaw.growing_smb_mount.trim()); const containerPath = growingRaw.growing_path || '/growing'; const mount = await probeGrowingPath(containerPath); @@ -117,6 +121,7 @@ router.get('/overview', async (req, res, next) => { // existing deploy uses this symlink — surface it for operator context. host_path: '/mnt/NVME/MAM/wild-dragon-growing', smb_url: growingRaw.growing_smb_url || '', + smb_mount: growingRaw.growing_smb_mount || '', promote_after_seconds: parseInt(growingRaw.growing_promote_after_seconds, 10) || 8, exists: mount.exists, writable: mount.writable, diff --git a/services/mam-api/src/scheduler.js b/services/mam-api/src/scheduler.js index 91ba0bb..143297e 100644 --- a/services/mam-api/src/scheduler.js +++ b/services/mam-api/src/scheduler.js @@ -192,13 +192,68 @@ async function enqueueNextOccurrence(schedule, client) { // ── Playout channel health + failover ──────────────────────────────────────── // Tick step 6. Reuses the same advisory lock so only one replica probes the -// sidecars. A missed probe is counted via last_heartbeat_at age: > 3 * -// TICK_INTERVAL means 3 consecutive misses. +// sidecars; multi-replica pings would just waste cycles. A missed probe is +// counted via last_heartbeat_at age: > 3 * TICK_INTERVAL means 3 consecutive +// misses. +// Persist the as-run compliance log for one channel from a sidecar /status +// payload. The sidecar reports the currently on-air item via currentItemId / +// currentClip / currentItemStartedAt (playout-manager.getStatus). We keep at +// most one "open" row (ended_at IS NULL) per channel: when the on-air item +// changes (or playout stops) we close the open row — stamping ended_at and a +// computed duration_s — and, if a new clip is on air, open a fresh row. // -// IMPORTANT: when last_heartbeat_at is NULL (channel just spawned, no -// successful tick yet), use updated_at as the grace anchor — otherwise the -// "0" fallback makes ageMs huge and the channel is instantly failover-killed -// before its first heartbeat can ever land. +// playout_as_run columns (migration 029): id, channel_id, asset_id, item_id, +// clip_name, started_at, ended_at, duration_s, result. +async function writeAsRun(client, channelId, engine) { + const currentItemId = engine && engine.currentItemId ? engine.currentItemId : null; + + // The currently-open as-run row for this channel, if any. + const { rows: openRows } = await client.query( + `SELECT id, item_id, started_at FROM playout_as_run + WHERE channel_id = $1 AND ended_at IS NULL + ORDER BY started_at DESC LIMIT 1`, + [channelId] + ); + const open = openRows[0] || null; + + // Same clip still on air → nothing to do. + if (open && currentItemId && open.item_id === currentItemId) return; + // Nothing on air and nothing open → nothing to do. + if (!open && !currentItemId) return; + + // Close the previous open row (clip changed, or playout stopped). + if (open) { + await client.query( + `UPDATE playout_as_run + SET ended_at = NOW(), + duration_s = EXTRACT(EPOCH FROM (NOW() - started_at)) + WHERE id = $1`, + [open.id] + ); + } + + // Open a new row for the clip now on air. Resolve the item's asset_id so the + // compliance log links back to the source asset even after the playlist item + // is later deleted. + if (currentItemId) { + let assetId = null; + try { + const { rows } = await client.query( + 'SELECT asset_id FROM playout_items WHERE id = $1', [currentItemId] + ); + if (rows.length > 0) assetId = rows[0].asset_id; + } catch (_) { /* item may have been deleted; log without asset link */ } + + await client.query( + `INSERT INTO playout_as_run + (channel_id, asset_id, item_id, clip_name, started_at, result) + VALUES ($1, $2, $3, $4, COALESCE($5::timestamptz, NOW()), 'played')`, + [channelId, assetId, currentItemId, engine.currentClip || null, + engine.currentItemStartedAt || null] + ); + } +} + async function playoutHealthTick(client) { let channels; try { @@ -223,10 +278,24 @@ async function playoutHealthTick(client) { await client.query( 'UPDATE playout_channels SET last_heartbeat_at = NOW() WHERE id = $1', [ch.id] ); + // As-run compliance log: the sidecar only tracks the on-air clip locally + // (playout-manager._reportAsRunStart). On every successful status poll we + // detect a clip change here and persist it to playout_as_run — close the + // previous open row and open a new one. Failures are swallowed so a logging + // hiccup never knocks a healthy channel into failover. + try { + const engine = await r.json().catch(() => null); + if (engine) await writeAsRun(client, ch.id, engine); + } catch (e) { + console.warn(`[scheduler] as-run write failed for ${ch.id}: ${e.message}`); + } } catch (err) { - const lastSeen = ch.last_heartbeat_at - ? new Date(ch.last_heartbeat_at).getTime() - : new Date(ch.updated_at).getTime(); + // When last_heartbeat_at is NULL (channel just spawned), fall back to + // updated_at (set to NOW() by spawnChannelSidecar). This prevents a + // brand-new channel from being failed over on the very first tick because + // epoch-0 age always exceeds TIMEOUT_MS. + const baseline = ch.last_heartbeat_at || ch.updated_at; + const lastSeen = baseline ? new Date(baseline).getTime() : Date.now(); const ageMs = Date.now() - lastSeen; if (ageMs < TIMEOUT_MS) continue; diff --git a/services/playout/Dockerfile b/services/playout/Dockerfile index de02af1..5b29ac2 100644 --- a/services/playout/Dockerfile +++ b/services/playout/Dockerfile @@ -1,4 +1,22 @@ # Wild Dragon Playout sidecar — CasparCG Server + Node AMCP control shim. +# +# CasparCG's mixer needs an OpenGL context. On a node with a real GPU we'd pass +# the device + driver through; for the headless / no-GPU case we run a virtual +# framebuffer (Xvfb) so the GL context initialises. The container is launched +# --privileged by mam-api (same as capture) so DeckLink / NDI hardware is +# reachable when present. +# +# CasparCG 2.4.x no longer ships a self-contained Linux tarball — the GitHub +# release provides either Ubuntu .deb packages or an "ubuntu22" zip that bundles +# the binary + its .so files under bin/ and lib/. We use the zip on an +# ubuntu:22.04 base so the bundled libs match the host glibc/abi, then install +# Node 20 from NodeSource on top. +# +# NDI + DeckLink SDKs are NOT redistributable. They are fetched at build time +# from a URL supplied as a build arg (mirror it into your own artifact store); +# the build still succeeds without it (NDI/DeckLink consumers simply won't be +# available — SRT/RTMP/test output still work). + FROM ubuntu:22.04 ARG CASPAR_VERSION=2.4.0-stable @@ -7,8 +25,16 @@ ARG NDI_SDK_URL= ENV DEBIAN_FRONTEND=noninteractive -# CEF (HTML producer) needs libnss3 + chromium runtime deps. Without these the -# server starts fine but SIGABRTs ~30s in when it lazy-inits CEF (NSS -8023). +# CasparCG 2.4 runtime deps + Xvfb for headless GL + CEF (HTML producer) deps + +# Node 20 (NodeSource). +# +# NOTE: we deliberately do NOT `apt-get install ffmpeg`. That package drags in +# ~80 transitive shared libraries (libav*, libx264, libdrm, libva, ...) that +# perturb CasparCG 2.4.0's runtime linking and make its headless startup abort +# with SIGABRT (exit 134) on nearly every launch. A self-contained STATIC +# ffmpeg binary (installed below) gives us the standalone CLI the preview +# re-muxer needs with ZERO new shared libs, keeping CasparCG's environment +# identical to the known-good image. RUN apt-get update && apt-get install -y --no-install-recommends \ ca-certificates curl unzip tar xz-utils gnupg \ xvfb libgl1-mesa-dri libglu1-mesa fonts-dejavu-core \ @@ -23,16 +49,34 @@ RUN apt-get update && apt-get install -y --no-install-recommends \ && apt-get update && apt-get install -y --no-install-recommends nodejs \ && rm -rf /var/lib/apt/lists/* +# ── Standalone STATIC ffmpeg CLI (for the HLS preview re-muxer) ─────────────── +# john van sickle's static build is fully self-contained (no shared-lib deps), +# so it can't perturb CasparCG's runtime linking. Override FFMPEG_URL to mirror +# this into your own artifact store if upstream availability is a concern. +ARG FFMPEG_URL=https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz +RUN set -eux; \ + curl -fsSL "$FFMPEG_URL" -o /tmp/ffmpeg.tar.xz; \ + mkdir -p /tmp/ffmpeg; \ + tar xJf /tmp/ffmpeg.tar.xz -C /tmp/ffmpeg --strip-components=1; \ + cp /tmp/ffmpeg/ffmpeg /tmp/ffmpeg/ffprobe /usr/local/bin/; \ + chmod +x /usr/local/bin/ffmpeg /usr/local/bin/ffprobe; \ + rm -rf /tmp/ffmpeg /tmp/ffmpeg.tar.xz; \ + /usr/local/bin/ffmpeg -version | head -1 + +# ── CasparCG Server (ubuntu22 zip bundle) ──────────────────────────────────── +# The zip extracts to /opt/casparcg_server with the binary at bin/casparcg and +# its bundled .so files under lib/ (added to LD_LIBRARY_PATH by entrypoint.sh). +# Symlink to /opt/casparcg so the config/entrypoint paths stay stable. WORKDIR /tmp/caspar RUN set -eux; \ - curl -fsSL "$CASPAR_URL" -o caspar.zip; \ - unzip -q caspar.zip -d /opt; \ - chmod +x /opt/casparcg_server/bin/casparcg /opt/casparcg_server/scanner 2>/dev/null || true; \ - ls /opt/casparcg_server/; \ - test -x /opt/casparcg_server/bin/casparcg; \ - ln -sfn /opt/casparcg_server /opt/casparcg; \ - echo "caspar binary: /opt/casparcg_server/bin/casparcg"; \ - cd /; rm -rf /tmp/caspar + curl -fsSL "$CASPAR_URL" -o caspar.zip; \ + unzip -q caspar.zip -d /opt; \ + chmod +x /opt/casparcg_server/bin/casparcg /opt/casparcg_server/scanner 2>/dev/null || true; \ + ls /opt/casparcg_server/; \ + test -x /opt/casparcg_server/bin/casparcg; \ + ln -sfn /opt/casparcg_server /opt/casparcg; \ + echo "caspar binary: /opt/casparcg_server/bin/casparcg"; \ + cd /; rm -rf /tmp/caspar RUN if [ -n "$NDI_SDK_URL" ]; then \ mkdir -p /opt/ndi-lib && \ diff --git a/services/playout/entrypoint.sh b/services/playout/entrypoint.sh index 264818c..7450abb 100644 --- a/services/playout/entrypoint.sh +++ b/services/playout/entrypoint.sh @@ -11,6 +11,8 @@ if [ -z "${DISPLAY:-}" ]; then done fi +# Ensure the HLS preview directory exists before the re-mux ffmpeg writes to it +# (mam-api serves /live//* from the shared media volume). if [ -n "${CHANNEL_ID:-}" ]; then mkdir -p "/media/live/${CHANNEL_ID}" fi diff --git a/services/playout/src/playout-manager.js b/services/playout/src/playout-manager.js index 95b856c..c846483 100644 --- a/services/playout/src/playout-manager.js +++ b/services/playout/src/playout-manager.js @@ -1,4 +1,6 @@ import { AmcpClient } from './amcp.js'; +import { spawn } from 'node:child_process'; +import { mkdirSync } from 'node:fs'; // Playout manager — owns one CasparCG channel's lifecycle inside this sidecar. // @@ -23,6 +25,12 @@ const MEDIA_ROOT = process.env.CASPAR_MEDIA_ROOT || '/media'; const CHANNEL_ID = process.env.CHANNEL_ID || ''; const HLS_DIR = CHANNEL_ID ? `${MEDIA_ROOT}/live/${CHANNEL_ID}` : ''; +// Loopback UDP port CasparCG's preview STREAM consumer publishes mpegts to, and +// the standalone ffmpeg re-muxer reads from. One CasparCG per sidecar, so a +// fixed port is fine; allow override for parallel local testing. +const PREVIEW_UDP_PORT = parseInt(process.env.PREVIEW_UDP_PORT || '9710', 10); +const PREVIEW_UDP_URL = `udp://127.0.0.1:${PREVIEW_UDP_PORT}`; + // CasparCG SEEK / LENGTH are in frames, not seconds. Capture standard is 59.94; // SD/film modes need their own values. Default 60000/1001 matches both // '1080p5994' and '1080i5994'. @@ -77,6 +85,8 @@ export class PlayoutManager { lastError: null, }; this._advanceTimer = null; + this._hlsProc = null; // standalone ffmpeg re-mux child process + this._hlsRestartTimer = null; } async _consumerCommand(outputType, cfg) { @@ -111,22 +121,38 @@ export class PlayoutManager { // Start the channel: bring up CasparCG's primary output consumer for the // target, plus a second FFMPEG consumer writing HLS for the UI preview // monitor (~4-6s lag, reuses capture's /live/ plumbing). + // + // The primary consumer failure is NON-FATAL. CasparCG can decode and route + // media through its pipeline even without an output consumer. This means: + // - NDI channels work (load/play/transport) even if libndi.so is absent. + // - SRT/RTMP channels work even if the destination URL is unreachable. + // - The HLS preview consumer is always attempted independently. + // + // state.consumerError is set when the primary consumer fails so the mam-api + // can surface a warning in the channel status without blocking operation. async startChannel({ outputType, outputConfig = {}, videoFormat = '1080p5994' }) { await this.amcp.waitReady(30000); - // Set the channel video mode, then attach the output consumer. + // Set the channel video mode first. try { await this.amcp.send(`SET ${CHANNEL} MODE ${videoFormat}`); } catch (err) { console.warn(`[playout] SET MODE failed (continuing): ${err.message}`); } - const consumer = await this._consumerCommand(outputType, outputConfig); - await this.amcp.send(`ADD ${CHANNEL} ${consumer}`); + // Primary output consumer — non-fatal. + let consumerError = null; + try { + const consumer = await this._consumerCommand(outputType, outputConfig); + await this.amcp.send(`ADD ${CHANNEL} ${consumer}`); + } catch (err) { + consumerError = err.message; + console.warn(`[playout] primary consumer ADD failed (continuing without output): ${err.message}`); + } + // HLS preview consumer — always attempt, independently non-fatal. if (HLS_DIR) { try { await this._addHlsConsumer(); console.log(`[playout] HLS preview at ${HLS_DIR}/index.m3u8`); } catch (err) { - // HLS preview is non-fatal — operators still get the on-air output. console.warn(`[playout] HLS preview consumer failed: ${err.message}`); } } @@ -137,37 +163,136 @@ export class PlayoutManager { this.state.videoFormat = videoFormat; this.state.fps = fpsFor(videoFormat); this.state.startedAt = new Date().toISOString(); - this.state.lastError = null; - console.log(`[playout] channel started output=${outputType} mode=${videoFormat} fps=${this.state.fps.toFixed(3)}`); + this.state.lastError = consumerError; + console.log(`[playout] channel started output=${outputType} mode=${videoFormat} fps=${this.state.fps.toFixed(3)}${consumerError ? ' ⚠ consumer: ' + consumerError : ''}`); return this.getStatus(); } - // Low-bitrate HLS for the web UI preview. Segments land in the shared media - // volume; the mam-api serves /live//* from there. + // HLS preview for the web UI confidence monitor. + // + // ── Why not CasparCG's own HLS (FILE/STREAM ".../index.m3u8") ────────────── + // CasparCG's bundled FFMPEG consumer muxes a BROKEN audio track into the HLS: + // ffprobe reports `aac, sample_rate=0` and ffmpeg decoding the playlist fails + // with "Invalid data ... abuffer: Value inf for parameter 'time_base' ... + // time_base 1/0". That corrupt audio prevents BOTH ffmpeg and hls.js from + // decoding, so the browser