Bug: Asset player stalls without buffering when seeking to end of clip #143

New issue

Closed

opened 2026-05-26 21:50:29 -04:00 by zgaetano · 0 comments

zgaetano commented

2026-05-26 21:50:29 -04:00

Owner

Stitching workaround deployed in a86c1c7 and verified live against the affected asset.

Root cause

RustFS returns empty bodies for ranged GETs whose START offset is past ~5.9 MB on single-file proxy MP4s. Confirmed via direct S3 SigV4 probe against broadcastmgmt.cloud/dragonmam:

Range	RustFS direct	Notes
HEAD object	size=11684523 etag=a490247d8df… parts=n/a	✓
full GET `bytes=0-`	11684523 b	✓
`bytes=5800000-5900000` (50%)	100001 b	✓
`bytes=8179166-8279166` (70%)	206 + Content-Range correct + 0 bytes	bug
`bytes=11674000-11684522` (EOF)	206 + 0 bytes	bug
Server-side `CopyObject` then re-read same offsets	same broken response, same etag	bug is in the range read path, storage is fine
Full GET then byte-window slicing	every offset matches the ETag MD5	confirms object is intact

So: storage holds the bytes, full GETs read them, but ranged GETs starting past the threshold silently return empty bodies with otherwise-correct 206 headers. Browser sees a partial-content response that contains nothing and stalls.

Fix (this commit)

services/mam-api/src/routes/assets.js → GET /:id/video:

HEAD the object first to learn the size + ETag.
No-Range / unparseable Range → plain bytes=0- GET (always works).
Parsed bytes=N-M whose end is below RUSTFS_RANGE_SAFE_START (default 5_500_000) → direct ranged GET, RustFS handles fine.
Anything reaching into the broken zone → stream from offset 0, drop bytes below start, stop at end. Browser sees a normal 206 + Content-Range; mam-api stays memory-flat; extra RustFS→mam-api bandwidth = (end+1 - requested_size) per seek.
Truly past-EOF → 416 with Cache-Control: no-store so the browser doesn't poison its cache and re-stall on the next seek.

Live verification (post-deploy)

no-range full GET              status=200 ttfb=300ms total=935ms bytes=11684523
bytes=0-1023                   status=206 ttfb=90ms  total=90ms  bytes=1024     cr=bytes 0-1023/11684523
bytes=5800000-5900000          status=206 ttfb=320ms total=324ms bytes=100001   cr=bytes 5800000-5900000/11684523
bytes=8179166-8279166  (70%)   status=206 ttfb=263ms total=268ms bytes=100001   cr=bytes 8179166-8279166/11684523
bytes=8179166- (open 70%)      status=206 ttfb=290ms total=398ms bytes=3505357  cr=bytes 8179166-11684522/11684523
near-EOF                       status=206 ttfb=287ms total=288ms bytes=10523    cr=bytes 11674000-11684522/11684523
past-EOF (should 416)          status=416 total=32ms bytes=0      cr=bytes */11684523 cache=no-store

All previously-stalled offsets now return correct byte counts.

Follow-up (v1.2.1)

Filed as separate work: switch the proxy worker to emit HLS (fmp4 segments) so every object stays well below the broken threshold and we stop paying the streaming-from-0 bandwidth tax on every seek. Helpers (segmentToHls, uploadDirectoryToS3) staged in this commit for the follow-up. A RustFS upstream bug report should also be opened with this evidence.

Stitching workaround deployed in a86c1c7 and verified live against the affected asset. ## Root cause RustFS returns empty bodies for ranged GETs whose START offset is past ~5.9 MB on single-file proxy MP4s. Confirmed via direct S3 SigV4 probe against `broadcastmgmt.cloud/dragonmam`: | Range | RustFS direct | Notes | |---|---|---| | HEAD object | size=11684523 etag=a490247d8df… parts=n/a | ✓ | | full GET `bytes=0-` | 11684523 b | ✓ | | `bytes=5800000-5900000` (50%) | 100001 b | ✓ | | `bytes=8179166-8279166` (70%) | **206 + Content-Range correct + 0 bytes** | bug | | `bytes=11674000-11684522` (EOF) | **206 + 0 bytes** | bug | | Server-side `CopyObject` then re-read same offsets | same broken response, same etag | bug is in the **range read path**, storage is fine | | Full GET then byte-window slicing | every offset matches the ETag MD5 | confirms object is intact | So: storage holds the bytes, full GETs read them, but ranged GETs starting past the threshold silently return empty bodies with otherwise-correct 206 headers. Browser sees a partial-content response that contains nothing and stalls. ## Fix (this commit) `services/mam-api/src/routes/assets.js → GET /:id/video`: - HEAD the object first to learn the size + ETag. - No-Range / unparseable Range → plain `bytes=0-` GET (always works). - Parsed `bytes=N-M` whose **end** is below `RUSTFS_RANGE_SAFE_START` (default 5_500_000) → direct ranged GET, RustFS handles fine. - Anything reaching into the broken zone → stream from offset 0, drop bytes below `start`, stop at `end`. Browser sees a normal 206 + Content-Range; mam-api stays memory-flat; extra RustFS→mam-api bandwidth = (end+1 - requested_size) per seek. - Truly past-EOF → 416 with `Cache-Control: no-store` so the browser doesn't poison its cache and re-stall on the next seek. ## Live verification (post-deploy) ``` no-range full GET status=200 ttfb=300ms total=935ms bytes=11684523 bytes=0-1023 status=206 ttfb=90ms total=90ms bytes=1024 cr=bytes 0-1023/11684523 bytes=5800000-5900000 status=206 ttfb=320ms total=324ms bytes=100001 cr=bytes 5800000-5900000/11684523 bytes=8179166-8279166 (70%) status=206 ttfb=263ms total=268ms bytes=100001 cr=bytes 8179166-8279166/11684523 bytes=8179166- (open 70%) status=206 ttfb=290ms total=398ms bytes=3505357 cr=bytes 8179166-11684522/11684523 near-EOF status=206 ttfb=287ms total=288ms bytes=10523 cr=bytes 11674000-11684522/11684523 past-EOF (should 416) status=416 total=32ms bytes=0 cr=bytes */11684523 cache=no-store ``` All previously-stalled offsets now return correct byte counts. ## Follow-up (v1.2.1) Filed as separate work: switch the proxy worker to emit HLS (fmp4 segments) so every object stays well below the broken threshold and we stop paying the streaming-from-0 bandwidth tax on every seek. Helpers (`segmentToHls`, `uploadDirectoryToS3`) staged in this commit for the follow-up. A RustFS upstream bug report should also be opened with this evidence.