From 2f1697b77bb1b74934ef5259fd2a1c2780d4e380 Mon Sep 17 00:00:00 2001 From: ZGaetano Date: Wed, 3 Jun 2026 10:48:57 -0400 Subject: [PATCH] feat(framecache): add implementation plan to docs --- docs/design/framecache/PLAN.md | 221 +++++++++++++++++++++++++++++++++ 1 file changed, 221 insertions(+) create mode 100644 docs/design/framecache/PLAN.md diff --git a/docs/design/framecache/PLAN.md b/docs/design/framecache/PLAN.md new file mode 100644 index 0000000..1cc37a8 --- /dev/null +++ b/docs/design/framecache/PLAN.md @@ -0,0 +1,221 @@ +# Unified Framecache — Implementation Plan + +## Context + +Replace the current named-FIFO-per-source architecture with a shared-memory +ring buffer (framecache) that fans raw video frames from any ingest source to +unlimited concurrent consumers with zero-copy reads. + +**Approved design:** docs/design/framecache/DESIGN.md +**Branch:** feat/unified-framecache +**Roadmap (out of scope here):** RDMA cross-node, AJA, growing-file-while-recording browser playback + +--- + +## Migration Strategy + +Ship in 5 phases. Each phase is independently deployable and leaves the system +in a working state. Existing recording workflows are unaffected until Phase 5 +cuts over. + +--- + +## Phase 1 — Framecache Container (foundation) + +**Goal:** Running framecache service with slot registry. No ingest writers yet. + +### 1.1 — Create `services/framecache/` directory structure + +``` +services/framecache/ + src/ + framecache.c # main — slot manager + HTTP API + slot.c / slot.h # shm ring buffer lifecycle + registry.c # /dev/shm/framecache/registry.json writer + http.c # lightweight HTTP server (libmicrohttpd) + client/ + fc_client.c / fc_client.h # consumer library + fc_client_node/ + binding.cc # Node.js N-API addon + binding.gyp + Dockerfile + CMakeLists.txt +``` + +### 1.2 — Shared memory layout (slot.h) + +Each slot lives at `/dev/shm/framecache/`: + +```c +#define FC_MAGIC 0x46524D43 // "FRMC" +#define FC_RING_DEPTH 120 // ~2s at 59.94fps +#define FC_HEADER_SIZE 4096 // 4KB header block + +typedef struct { + uint32_t magic; + uint32_t version; // = 1 + uint32_t width; + uint32_t height; + uint32_t fps_num; + uint32_t fps_den; + uint32_t pixel_format; // FC_PIX_UYVY422 = 0 + uint32_t frame_size; // width * height * 2 + uint32_t ring_depth; // = FC_RING_DEPTH + _Atomic uint64_t write_cursor; // monotonically increasing frame index + _Atomic uint64_t dropped_frames; + uint8_t _pad[FC_HEADER_SIZE - 48]; +} fc_header_t; + +typedef struct { + uint64_t pts_us; + uint64_t wall_us; + uint32_t size; + uint8_t data[]; // frame_size bytes +} fc_frame_t; +``` + +Semaphore: `sem_open("/framecache--write", ...)` — posted by writer +on each new frame, consumers `sem_timedwait` on it. + +### 1.3 — HTTP API (port 7435) + +``` +POST /slots body: {slot_id, width, height, fps_num, fps_den, source_type} + creates shm region, writes registry entry + 201 {slot_id, shm_path, sem_name} + +GET /slots 200 [{slot_id, width, height, fps_num, fps_den, + source_type, write_cursor, dropped_frames, + current_fps}] + +GET /slots/:id 200 slot detail +DELETE /slots/:id destroys shm + semaphore, removes registry entry, 204 +GET /health 200 {status: "ok"} +``` + +### 1.4 — Registry file + +Written to `/dev/shm/framecache/registry.json` on every slot create/delete. + +### 1.5 — Dockerfile + +```dockerfile +FROM debian:bookworm +RUN apt-get update && apt-get install -y \ + build-essential cmake libmicrohttpd-dev \ + && rm -rf /var/lib/apt/lists/* +COPY . /src +RUN cmake -S /src -B /build -DCMAKE_BUILD_TYPE=Release \ + && cmake --build /build -j$(nproc) +EXPOSE 7435 +CMD ["/build/framecache"] +``` + +### 1.6 — docker-compose.worker.yml addition + +```yaml +framecache: + build: ./services/framecache + ipc: host + shm_size: '60gb' + environment: + FC_SHM_SIZE: ${FC_SHM_SIZE:-64424509440} + FC_PORT: 7435 + ports: + - "7435:7435" + volumes: + - /dev/shm:/dev/shm + restart: unless-stopped +``` + +### 1.7 — Consumer library (fc_client.c) + +```c +fc_slot_t *fc_open(const char *slot_id); +int fc_read_frame(fc_slot_t *slot, fc_frame_t **out, uint64_t timeout_ms); +void fc_close(fc_slot_t *slot); +``` + +**Commit:** `feat(framecache): phase 1 — framecache container + consumer library` + +--- + +## Phase 2 — Deltacast Bridge writes to framecache + +**Goal:** deltacast-bridge writes frames to framecache shm instead of named FIFOs. +Legacy FIFO path kept as compile-time fallback (`-DLEGACY_FIFO=ON`) until Phase 5. + +On signal lock: +1. POST /slots to framecache HTTP API +2. shm_open + mmap the slot +3. Video thread writes frame into ring, advances write_cursor atomically, sem_post +4. Audio: keeps writing to audio FIFO (unchanged) +5. On shutdown: DELETE /slots/:id + +**Commit:** `feat(framecache): phase 2 — deltacast-bridge writes to shm` + +--- + +## Phase 3 — Blackmagic DeckLink Bridge + +**Goal:** New decklink-bridge C program mirrors deltacast-bridge, replaces +ffmpeg -f decklink direct path. + +- Uses IDeckLinkIterator to enumerate devices +- VideoInputFrameArrived callback calls fc_write_frame +- Registers slot on signal lock, deregisters on shutdown +- Audio stays in FIFO (same as deltacast) + +**Commit:** `feat(framecache): phase 3 — decklink-bridge writes to shm` + +--- + +## Phase 4 — capture-manager reads from framecache + +**Goal:** Enables simultaneous growing + proxy + HLS from one SDI input. + +- Node.js N-API addon wrapping fc_open/fc_read_frame/fc_close +- capture-manager opens THREE fc_client handles per slot (own cursor each): + 1. Growing/master ffmpeg feed + 2. Proxy ffmpeg feed + 3. HLS preview ffmpeg feed +- Each gets a separate rawvideo pipe to ffmpeg +- Growing MXF workflow (raw2bmx orchestrator) completely unchanged + +**Commit:** `feat(framecache): phase 4 — capture-manager reads from framecache` + +--- + +## Phase 5 — Network ingest (RTMP/SRT) into framecache + +**Goal:** RTMP and SRT sources decoded to raw UYVY422, written into framecache slots. + +- net_ingest process per source: ffmpeg decodes to rawvideo, writes to slot +- capture-manager waits for slot, same fc_client consumer pattern +- Remove legacy FIFO code once all paths go through framecache + +**Commit:** `feat(framecache): phase 5 — network ingest via framecache` + +--- + +## Hardware / Deployment + +| Node | RAM | /dev/shm | FC_SHM_SIZE | +|------|-----|----------|-------------| +| Baratheon | 251GB | 126GB | 60GB | +| zampp1 | 93GB | 47GB | 40GB | +| zampp2 | 18GB (upgrade) | 9.4GB | 8GB | + +Ring buffer per 1080p59.94 source: ~494MB (120 frames × 4.1MB) + +All recorder sidecars require `ipc: host`. + +--- + +## Roadmap (not in this branch) + +- Audio in framecache shm +- RDMA cross-node slot replication +- AJA hardware support +- Growing-file-while-recording browser HLS playback +- Mastercontrol/playout consumer