Backend: multer.memoryStorage holds 500 MB parts in RAM — concurrent uploads OOM the API #120

New issue

Closed

opened 2026-05-26 18:21:19 -04:00 by zgaetano · 1 comment

zgaetano commented

2026-05-26 18:21:19 -04:00

Owner

Fixed in 04ce096. services/mam-api/src/routes/upload.js now uses multer.diskStorage writing to UPLOAD_TMP_DIR (defaults to os.tmpdir()/df-uploads). /part and /simple stream the on-disk file to S3 with fs.createReadStream and unlink the tmp file in finally. Concurrent 500 MB uploads no longer hold their bytes in RAM.

Fixed in 04ce096. `services/mam-api/src/routes/upload.js` now uses `multer.diskStorage` writing to `UPLOAD_TMP_DIR` (defaults to `os.tmpdir()/df-uploads`). `/part` and `/simple` stream the on-disk file to S3 with `fs.createReadStream` and unlink the tmp file in `finally`. Concurrent 500 MB uploads no longer hold their bytes in RAM.

~~zgaetano referenced this issue 2026-05-26 18:22:49 -04:00~~

Tracker: 2026-05-26 audit session — 27 bugs logged (#100-#127) #128

zgaetano commented

2026-05-26 19:32:01 -04:00

Author

Owner

Fix Plan — #120 multer.memoryStorage OOM on concurrent uploads

Root cause: upload.js and assets.js use multer.memoryStorage() with 500MB cap. Each concurrent upload chunk lives in heap. 3 operators × 4GB files = ~12GB → OOM.

Fix — switch to disk storage + stream to S3:

// before:
multer({ storage: multer.memoryStorage(), limits: { fileSize: 500e6 } })

// after:
multer({
  storage: multer.diskStorage({ destination: "/tmp/uploads" }),
  limits: { fileSize: 500e6 }
})
// then stream file from disk to S3, unlink after success

Better: use @aws-sdk/lib-storage Upload class with createReadStream() for streaming multipart direct to S3.

Files: src/routes/upload.js, src/routes/assets.js
Effort: ~2h
**Priority: P1 — OOM risk

## Fix Plan — #120 multer.memoryStorage OOM on concurrent uploads **Root cause:** `upload.js` and `assets.js` use `multer.memoryStorage()` with 500MB cap. Each concurrent upload chunk lives in heap. 3 operators × 4GB files = ~12GB → OOM. **Fix — switch to disk storage + stream to S3:** ```js // before: multer({ storage: multer.memoryStorage(), limits: { fileSize: 500e6 } }) // after: multer({ storage: multer.diskStorage({ destination: "/tmp/uploads" }), limits: { fileSize: 500e6 } }) // then stream file from disk to S3, unlink after success ``` Better: use `@aws-sdk/lib-storage` `Upload` class with `createReadStream()` for streaming multipart direct to S3. **Files:** `src/routes/upload.js`, `src/routes/assets.js` **Effort:** ~2h **Priority: P1 — OOM risk

zgaetano referenced this issue from a commit

2026-05-26 22:06:25 -04:00

chore: 1.2 ship-prep sweep — close 38 issues

zgaetano closed this issue

2026-05-26 22:08:25 -04:00