zgaetano/datarhei-dragonfork-core
1
0
Fork 0
Code Issues 2 Pull requests Projects Releases 4 Packages Wiki Activity Actions

M4: WebRTC server-hop latency p95 gate #9

Closed
zgaetano wants to merge 1 commit from m4-latency-gate into m4-ci-and-tooling
pull from: m4-latency-gate
merge into: zgaetano:m4-ci-and-tooling
Conversation 1 Commits 1 Files changed 3 +345 -5
zgaetano commented 2026-05-03 08:19:20 -04:00
Owner
Copy link

Second half of M4: a CI-enforced latency p95 gate. Stacks on PR #8 so the workflow file gets the new job in one merge.

What it measures

End-to-end RTP arrival latency from corewebrtc.Source ingest through Pion DTLS-SRTP to a subscriber's track.ReadRTP(). Synthetic publisher embeds time.Now().UnixNano() in each RTP payload; subscriber diffs against time.Now() on arrival.

Exercises every link the egress code path can regress:

  • Source.readLoop UDP read + RTP unmarshal
  • subscriber fan-out
  • forwardRTPSplit goroutine
  • Pion TrackLocalStaticRTP.WriteRTP
  • DTLS-SRTP encrypt
  • ICE socket write
  • subscriber DTLS-SRTP decrypt
  • TrackRemote.ReadRTP unmarshal

What it doesn't measure (and why)

The design's §7 calls for true glass-to-glass latency: drawtext frame counter on the publisher, decode-side pixel sampling on the subscriber, p95 < 300ms (RTMP) / < 200ms (SRT). Implementing that in pure Go would need a cgo H.264 decoder or an FFmpeg-as-sidecar pipe — a much bigger lift for marginal regression-detection value, since encode/decode latency is fixed by the codec stack and isn't moved by Core code changes.

The server-hop measurement captures everything Core code can actually regress.

Threshold

p95 < 50 ms. Locally observed on a quiet host:

p50 = 110 µs
p95 = 237 µs
p99 = 318 µs

So the gate is ~200× headroom — generous enough to absorb CI runner noise without false alarms, tight enough to catch a real slowdown.

Implementation notes

  • Build-tagged (//go:build latency) so it doesn't run in the default go test ./... invocation.
  • 1000 packets at 60 Hz → ~17s wall-clock per run.
  • latencySamples uses a sync.Mutex around the slice append because the receive goroutine and test goroutine race otherwise (vet caught this on the first draft).
  • New latency-gate job in .forgejo/workflows/test.yml, parallel with test and webrtc-smoke, depends on lint-and-vet.
  • Documented in test/TESTING.md with expected numbers and the rationale.

Files

app/webrtc/latency_test.go    | 296 +++++++++++++++++++++++++++++++++++++++
.forgejo/workflows/test.yml   |  24 ++++
test/TESTING.md               |  37 ++++--

Co-authored with Claude Opus 4.7.

Second half of M4: a CI-enforced latency p95 gate. Stacks on PR #8 so the workflow file gets the new job in one merge. ## What it measures End-to-end RTP arrival latency from `corewebrtc.Source` ingest through Pion DTLS-SRTP to a subscriber's `track.ReadRTP()`. Synthetic publisher embeds `time.Now().UnixNano()` in each RTP payload; subscriber diffs against `time.Now()` on arrival. Exercises every link the egress code path can regress: - `Source.readLoop` UDP read + RTP unmarshal - subscriber fan-out - `forwardRTPSplit` goroutine - Pion `TrackLocalStaticRTP.WriteRTP` - DTLS-SRTP encrypt - ICE socket write - subscriber DTLS-SRTP decrypt - `TrackRemote.ReadRTP` unmarshal ## What it doesn't measure (and why) The design's §7 calls for true glass-to-glass latency: `drawtext` frame counter on the publisher, decode-side pixel sampling on the subscriber, p95 < 300ms (RTMP) / < 200ms (SRT). Implementing that in pure Go would need a cgo H.264 decoder or an FFmpeg-as-sidecar pipe — a much bigger lift for marginal regression-detection value, since encode/decode latency is fixed by the codec stack and isn't moved by Core code changes. The server-hop measurement captures everything Core code can actually regress. ## Threshold `p95 < 50 ms`. Locally observed on a quiet host: ``` p50 = 110 µs p95 = 237 µs p99 = 318 µs ``` So the gate is ~200× headroom — generous enough to absorb CI runner noise without false alarms, tight enough to catch a real slowdown. ## Implementation notes - Build-tagged (`//go:build latency`) so it doesn't run in the default `go test ./...` invocation. - 1000 packets at 60 Hz → ~17s wall-clock per run. - `latencySamples` uses a `sync.Mutex` around the slice append because the receive goroutine and test goroutine race otherwise (vet caught this on the first draft). - New `latency-gate` job in `.forgejo/workflows/test.yml`, parallel with `test` and `webrtc-smoke`, depends on `lint-and-vet`. - Documented in `test/TESTING.md` with expected numbers and the rationale. ## Files ``` app/webrtc/latency_test.go | 296 +++++++++++++++++++++++++++++++++++++++ .forgejo/workflows/test.yml | 24 ++++ test/TESTING.md | 37 ++++-- ``` Co-authored with Claude Opus 4.7.
zgaetano added 1 commit 2026-05-03 08:19:20 -04:00
ci(webrtc): server-hop latency p95 gate
Some checks failed
ci / vet + build (push) Successful in 9m54s
Details
ci / vet + build (pull_request) Successful in 9m49s
Details
ci / race tests (push) Failing after 8m1s
Details
ci / WebRTC smoke (5-viewer fanout) (push) Successful in 9m45s
Details
ci / WebRTC latency p95 gate (push) Successful in 10m3s
Details
ci / race tests (pull_request) Failing after 7m59s
Details
ci / WebRTC smoke (5-viewer fanout) (pull_request) Successful in 9m45s
Details
ci / WebRTC latency p95 gate (pull_request) Successful in 10m4s
Details
b7afd0f08a 
Adds an end-to-end RTP-arrival latency probe that runs as a dedicated
CI job and asserts p95 < 50ms.

Implementation
--------------
A build-tagged test (-tags latency, off by default) sends 1000
synthetic RTP packets at 60Hz into corewebrtc.Source and reads them
back via a Pion subscriber's track.ReadRTP(). Each packet's payload
starts with the publisher's UnixNano send time; the subscriber diffs
against time.Now() at arrival and accumulates p50/p95/p99.

This exercises every link of the egress hop: Source UDP read,
subscriber fan-out, forwardRTPSplit, Pion's TrackLocalStaticRTP
write, DTLS-SRTP encrypt, ICE socket write, decrypt at the
subscriber, RTP unmarshal at ReadRTP. Pure server-side; no FFmpeg
or codecs involved.

Why not glass-to-glass
----------------------
The design's §7 calls for FFmpeg drawtext frame counters + decode-
side pixel sampling, p95<300ms RTMP / <200ms SRT. Implementing that
in pure Go needs a cgo H.264 decoder or an FFmpeg sidecar pipe — a
significantly bigger lift for a marginal regression-detection win
(encode/decode latency is roughly fixed by the codec stack and
isn't moved by Core code changes). The server-hop measurement
captures everything Core code can actually regress.

Threshold
---------
50ms p95. Locally observed on a quiet host:
  p50=110µs, p95=237µs, p99=318µs.
The 50ms gate is ~200x headroom — generous enough to absorb CI
runner noise without false alarms, tight enough to catch a real
slowdown.

Race-clean: latencySamples uses a sync.Mutex around the slice append
(initial draft had a slice racing with the receive goroutine; vet
caught it).

Documented in test/TESTING.md and wired to .forgejo/workflows/test.yml
as the latency-gate job (depends on lint-and-vet, parallel with test
and webrtc-smoke).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
zgaetano commented 2026-05-03 08:28:58 -04:00
Author
Owner
Copy link

Merged into main via direct push as part of the v0.1.0-dragonfork release. Branch commits are reachable from main; closing this PR. Release: https://forge.wilddragon.net/zgaetano/datarhei-dragonfork-core/releases/tag/v0.1.0-dragonfork

Merged into `main` via direct push as part of the v0.1.0-dragonfork release. Branch commits are reachable from main; closing this PR. Release: https://forge.wilddragon.net/zgaetano/datarhei-dragonfork-core/releases/tag/v0.1.0-dragonfork
zgaetano closed this pull request 2026-05-03 08:28:58 -04:00
Some checks failed
ci / vet + build (push) Successful in 9m54s
Details
ci / vet + build (pull_request) Successful in 9m49s
Details
ci / race tests (push) Failing after 8m1s
Details
ci / WebRTC smoke (5-viewer fanout) (push) Successful in 9m45s
Details
ci / WebRTC latency p95 gate (push) Successful in 10m3s
Details
ci / race tests (pull_request) Failing after 7m59s
Details
ci / WebRTC smoke (5-viewer fanout) (pull_request) Successful in 9m45s
Details
ci / WebRTC latency p95 gate (pull_request) Successful in 10m4s
Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: zgaetano/datarhei-dragonfork-core#9
No description provided.
Delete branch "m4-latency-gate"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?