rainbow-dragon/docs/plans/001-implementation-plan.md

# Forge — Implementation Plan

On-set live grading. Pushes CDL/LUT looks over IP directly to Alexa 35 / RED DSMC3 / Venice 2. Node-stack grade engine, Tangent Wave panel, auto clip logging from camera metadata, DeckLink SDI I/O, shot library + post deliverables.

## Ground rules

- Swift 6.0.3, SwiftPM monorepo at `forge/`. Toolchain: `/opt/swift/usr/bin` (add to PATH).
- TDD: test first, watch fail, implement, watch pass. Every task.
- Commit after every green task. Message style: `core: add CDL node math`, `driver-arri: parse rec state`.
- Dev box is Linux; product targets macOS. All hardware (cameras, DeckLink, Tangent, Metal) sits behind Swift protocols. Linux gets mock/sim implementations — these ARE the test doubles. macOS real implementations land later behind `#if os(macOS)` and are the only untestable-here code.
- YAGNI: no speculative config, no plugin systems, no premature abstraction beyond the hardware seams listed.

## Package layout (SPM targets, one repo)

```
forge/
  Package.swift
  Sources/
    ForgeColor/        // color math: CDL, curves, LUT, color spaces, flattening
    ForgeGrade/        // node stack, grade documents, serialization, gang/link
    ForgeCamera/       // CameraDriver protocol, connection supervisor, discovery
    ForgeCameraARRI/   // Alexa 35 REST driver
    ForgeCameraRED/    // DSMC3 RCP2 driver
    ForgeCameraSony/   // Venice 2 CNA driver
    ForgeSim/          // simulated cameras (in-process HTTP/TCP servers) for tests + demo
    ForgeLogger/       // rec-state watcher, shot record creation
    ForgeLibrary/      // SQLite project/day/shot store
    ForgeExport/       // CDL/CCC/CUBE/ALE/CSV/EDL writers
    ForgePanel/        // panel abstraction (TIPC protocol layer, mappable controls)
    ForgeVideo/        // video I/O abstraction (frame source/sink protocols)
    forge-cli/         // headless CLI: run engine, connect sim/real cameras, script grades
  Tests/ (mirror per target)
  App/ (macOS-only Xcode project, later phase, not buildable on Linux)
```

Dependencies kept minimal: swift-nio (TCP/HTTP servers+clients for drivers and sims), sqlite3 via system module (no GRDB on Linux — thin hand-rolled DAO), swift-argument-parser (CLI). Nothing else.

## Color/data model decisions (fixed now)

- Working space: linear float RGB internally. Input transforms: LogC4, Log3G10+RWG, S-Log3+SGamut3.Cine → linear; output transform for preview: Rec709 (2.4 gamma). Published coefficient sets from vendor whitepapers, encoded as tested constants.
- CDL math per ASC spec: `out = (in * slope + offset)^power`, sat per Rec709 luma weights (0.2126/0.7152/0.0722) — matches ASC CDL 1.2.
- Node stack: ordered `[GradeNode]`, kinds: `.inputTransform`, `.cdl`, `.saturation`, `.curves` (4x cubic spline master+RGB), `.lut3d`, `.outputTransform`. Enable/bypass per node.
- Flattening: evaluate stack over 33³ (default) or 65³ lattice in camera-native log space → single 3D LUT. If node[0] is CDL and target camera supports native CDL, split: CDL stays live-tweakable, remainder baked.
- Grade snapshot: full stack serialized JSON, stored per shot. Deterministic encoding (sorted keys) so snapshots diff cleanly.

## CameraDriver seam (fixed now)

```swift
public protocol CameraDriver: Actor {
  var capabilities: CameraCapabilities { get }        // nativeCDL, lut3dSizes, metadataFields...
  func connect() async throws
  func disconnect() async
  var state: AsyncStream<CameraState> { get }          // connection, rec state, clip name, metadata
  func push(look: FlattenedLook) async throws          // CDL and/or LUT per capabilities
}
```

`ConnectionSupervisor` owns retry/backoff/health per slot. `ForgeSim` implements vendor-faithful fake endpoints (ARRI-style REST JSON, RCP2-style TCP framing, CNA-style) so drivers are tested against wire formats, not hand-waved mocks.

---

## Phase 1 — Core color + grade engine (all-Linux, pure math)

### Task 1.1: Repo + package skeleton
- `swift package init`, targets above (empty), CI-style `swift test` green.
- Commit: `chore: package skeleton`.

### Task 1.2: ForgeColor — CDL
- Tests: identity CDL passes pixels through; known slope/offset/power vectors (hand-computed); sat=0 gives luma gray; negative-input power clamp behavior per ASC (clamp to 0 pre-power).
- Impl: `struct CDL { slope, offset, power: SIMD3<Float>; sat: Float }`, `apply(_ rgb: SIMD3<Float>) -> SIMD3<Float>`.

### Task 1.3: ForgeColor — transfer functions
- Tests: round-trip lin→log→lin < 1e-4 for LogC4, Log3G10, S-Log3; spot-check published anchor values (e.g. LogC4 18% gray encodes to documented code value).
- Impl: encode/decode pairs + gamut matrices (RWG→Rec709 etc. as 3x3s).

### Task 1.4: ForgeColor — 3D LUT
- Tests: identity lattice is no-op; trilinear interp matches direct eval within tolerance; CUBE parse/write round-trip; malformed CUBE rejected.
- Impl: `Lut3D` (size, flat float array), trilinear sample, `.cube` reader/writer.

### Task 1.5: ForgeColor — curves
- Tests: identity spline no-op; monotonic control points produce monotonic output; endpoint pinning.
- Impl: Catmull-Rom/cubic hermite spline, master + per-channel.

### Task 1.6: ForgeGrade — node stack eval
- Tests: empty stack = identity; ordering matters (CDL→LUT ≠ LUT→CDL); bypass flag skips node; full stack on known pixel vectors.
- Impl: `GradeStack.evaluate(_:)` composing ForgeColor prims.

### Task 1.7: ForgeGrade — flattening
- Tests: flattened 33³ LUT of a stack matches direct eval within 1/1024 max err on random samples in-gamut; CDL-split mode: node[0] CDL excluded from bake, returned separately; deterministic output.
- Impl: `Flattener.flatten(stack:latticeSize:splitLeadingCDL:) -> FlattenedLook`.

### Task 1.8: ForgeGrade — serialization + snapshots
- Tests: JSON round-trip stack equality; sorted-key determinism (byte-equal re-encode); unknown-node-kind decode fails loudly.
- Impl: Codable with stable encoder config.

## Phase 2 — Camera seam + ARRI driver + sim

### Task 2.1: ForgeCamera — protocol, capabilities, supervisor
- Tests: supervisor retries with backoff against a driver that fails N times then connects; state stream publishes disconnect on driver error; slot registry add/remove.
- Impl: as seam above + `CameraSlot`, `SlotRegistry`.

### Task 2.2: ForgeSim — ARRI-style simulator
- In-process NIO HTTP server mimicking Alexa 35 REST shape: auth-less JSON endpoints for system info, rec state, clip name, EI/WB/tint/TC, look upload accept + store. Scriptable: test can flip rec state, change clip name.
- Tests: sim serves documented-shape JSON; look upload stored + retrievable.

### Task 2.3: ForgeCameraARRI — driver against sim
- Tests (all against sim): connect + capability report; metadata poll populates state stream; rec start/stop events emitted with clip name; `push(look:)` uploads flattened LUT + CDL, sim receives correct payload; reconnect after sim restart.
- Impl: REST client, poll loop (200ms metadata, 100ms rec state), ALF4-equivalent payload assembly. NOTE: exact ARRI REST paths verified against public API docs when macOS/hardware phase starts; sim encodes best-documented shape and driver keeps paths in one constants file for easy correction.

### Task 2.4: forge-cli MVP
- `forge sim start`, `forge connect <ip> --driver arri`, `forge look load <file.json>`, `forge look push`, `forge status`. Wire supervisor + driver + grade engine.
- Tests: CLI integration test — start sim, connect, push look, assert sim received it.

## Phase 3 — Clip logger + library + exports

### Task 3.1: ForgeLibrary — SQLite store
- Tests: create project/day/shot CRUD; grade snapshot blob round-trip; migration table v1; concurrent writer safety (serialized queue).
- Impl: system sqlite3 module, thin DAO, WAL mode.

### Task 3.2: ForgeLogger — auto shot records
- Tests: rec-start event from driver → shot row with clip name, TC-in, metadata snapshot, current grade snapshot; rec-stop → TC-out/duration; offline camera → manual shot API; duplicate rec-start debounce.
- Impl: subscribes slot state streams, writes via ForgeLibrary.

### Task 3.3: ForgeExport — CDL/CCC/CUBE
- Tests: golden-file comparisons for .cdl XML, .ccc multi-clip, .cube from snapshot; re-import parses equal.

### Task 3.4: ForgeExport — ALE/CSV/EDL
- Tests: ALE header/column golden file with camera metadata fields; CMX3600 EDL with CDL comments (`* ASC_SOP`/`* ASC_SAT` lines) parse-checked; CSV escaping.

### Task 3.5: CLI: `forge export --day <d> --format ale|edl|cdl|cube|csv`
- Integration test: sim session → 3 auto-logged shots → export all formats → golden compare.

## Phase 4 — RED + Sony drivers

### Task 4.1: ForgeSim RCP2-style TCP sim + ForgeCameraRED
- RCP2 is length-prefixed JSON-ish over TCP. Sim implements handshake, param get/set/notify for rec state, clip name, IPP2 CDL slots, LUT upload. Driver tests mirror 2.3 set. Native CDL capability = true → exercises CDL-split flatten path end-to-end.

### Task 4.2: ForgeSim CNA-style sim + ForgeCameraSony
- Venice 2 capability-narrow: LUT push + metadata read; nativeCDL=false. Tests mirror 2.3. Capability flags verified: pushing CDL-split look to Sony driver auto-falls-back to full bake (test this explicitly).

### Task 4.3: Multicam gang
- Tests: gang A+B → edit on A pushes to both, each flattened per own camera capabilities/log space; ungang stops propagation; per-slot override survives gang edits.

## Phase 5 — Panel + video seams

### Task 5.1: ForgePanel — TIPC protocol layer
- Tangent Hub speaks TIPC over TCP (port 64246) — protocol layer fully testable on Linux against a scripted fake Hub. Tests: handshake/registration XML map, encoder delta → CDL param change with sensitivity curve, button → command dispatch, display text writeback (panel shows param values), focus-follows-slot.
- Impl: NIO TCP client, control-map model (trackballs→SOP, rings→master, knobs→sat/contrast/node, buttons→slot/bypass/still/new-shot).

### Task 5.2: ForgeVideo — frame source/sink seam
- Protocols: `FrameSource` / `FrameSink` (pixel buffer + timecode). Linux impl: file-based test source (reads DPX/PNG frames), null sink. Tests: grade-stack applied to test frame matches golden graded frame; still-grab writes correct file, attaches to shot.
- DeckLink real impl deferred to macOS phase; seam proven here.

### Task 5.3: Debounced push pipeline
- Tests: rapid param changes (simulated panel spins) → preview eval every change, camera push coalesced to ≤1 per 150ms, final state always pushed (trailing edge); per-camera independent debounce.

## Phase 7 — Offload + verification

### Task 7.1: ForgeOffload — hashing
- Tests: xxHash64 + MD5 known-vector tests; streaming hash of large file matches whole-file hash; chunked reader.
- Impl: xxHash64 (pure Swift, spec vectors), MD5 via system, `HashingReader`.

### Task 7.2: ForgeOffload — copy engine
- Tests: source tree → 2 destinations, byte-identical, hash-verified via post-copy re-read; interrupted copy resumes (partial file detected, re-copied); collision policy (exists+same hash = skip, exists+diff hash = error).
- Impl: `OffloadJob` (source, [destinations]), sequential copy w/ streaming hash, verify pass re-reads destination.

### Task 7.3: ForgeOffload — MHL v2 manifest
- Tests: golden-file MHL XML (ASC MHL spec shape: hashes, paths, sizes, creator info); manifest verifies against tree; tamper detection (flipped byte → verify fail).
- Impl: MHL writer + verifier.

### Task 7.4: Offload reports + library link
- Tests: HTML/CSV report golden; offloaded clip matched to shot record by clip name.
- CLI: `forge offload <src> --to <dst1> --to <dst2>`, `forge verify <mhl>`.

## Phase 8 — Proxy generation

### Task 8.1: ForgeProxy — ffmpeg wrapper
- Tests: command-line assembly golden (ProRes/DNxHR/H.264 presets, scale, lut3d filter path quoting, TC+clipname drawtext burn-in); process runner against `true`/fake-ffmpeg script asserting argv; real ffmpeg smoke test if binary present (skip otherwise).
- Impl: `ProxyPreset`, `FFmpegCommandBuilder`, `ProxyJob` queue (serial v1).

### Task 8.2: Graded proxies
- Tests: shot grade snapshot → flattened .cube written to temp → lut3d filter arg references it; ungraded fallback when no snapshot.
- CLI: `forge proxy --day <d> --preset prores-proxy --burn-look`.

### Task 8.3: Watch folder
- Tests: new file in watched dir → job enqueued once (debounce partial writes via size-stable check); already-processed skip list.

## Phase 6 — macOS (requires Mac hardware; out of scope on this box, scaffolded only)

- SwiftUI app shell (3-zone layout per design), Metal preview renderer (same GradeStack→shader codegen or LUT texture sampling), DeckLink SDK C++ bridge implementing FrameSource/Sink, Tangent Hub local connection, real-camera path verification (ARRI paths const file, RED SDK agreement, Venice protocol docs).
- Everything above compiles as dependencies; app layer is thin.

## Definition of done per task
1. Tests written first, seen failing.
2. Impl minimal to pass.
3. `swift test` fully green.
4. Commit.

## Risks carried
- ARRI/RED/Sony exact wire details need doc/hardware verification — isolated in per-driver constants files + sims, cheap to correct.
- Venice grade path may stay LUT-only. Capability system absorbs this.
- Metal/DeckLink/Tangent-USB untestable here — protocol/TIPC layer testable, seams proven with fakes.