Commit graph

3 commits

Author SHA1 Message Date
cf1fe136d0 fix(worker): route import queue to a consumer + refresh stale yt-dlp
YouTube imports were stuck in `ingesting` forever: after the capability-
routing split (fdec2e3), every worker's WORKER_QUEUES enumerated an explicit
list that omitted `import`, so index.js's `want('import')` was false on every
worker and nothing consumed the import queue. Add `import` to worker-p4
(heavy/primary worker; import is concurrency-1 and network-bound, one consumer
is enough).

Second defect that would have surfaced once routed: the baked yt-dlp came from
the distro package (Ubuntu 22.04 apt = 2022.04.08, ~4 years stale), which
current YouTube rejects. Pull the latest self-contained yt-dlp release binary
at image build instead (GPU image: yt-dlp_linux; alpine image: python zipapp).
Rebuild the worker image to refresh.

Verified on zampp1: import jobs drain ingesting -> processing -> ready,
failed=0, yt-dlp 2026.03.17.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-05-29 19:51:27 -04:00
9ad88e4df4 feat(ingest): YouTube importer — paste link, asset travels normal pipeline
Adds Ingest → YouTube. UI takes a URL + project, API enqueues a BullMQ
"import" job, worker shells out to yt-dlp, lands the MP4 in S3 at the
same originals/{assetId}/... path uploads use, then hands off to the
existing proxy queue. Imported assets share one lifecycle with uploads
from that point on.

Worker container picks up yt-dlp + python3 (apk on alpine, apt on the
GPU variant). The new 'import' queue is registered in jobs.js so it
appears in the Jobs SSE stream and retry/delete work for free.

Spec: docs/superpowers/specs/2026-05-23-youtube-importer-design.md

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23 16:05:41 -04:00
167bf25443 add services/worker/Dockerfile 2026-04-07 21:58:18 -04:00