Compare commits

...

2 commits

Author SHA1 Message Date
a7ef0397e1 fix(web-ui): show live download % on YouTube import bar
The import queue row's progress bar read only `r.progress`, which pollRow
never updated numerically — it tracked asset.status (ingesting/processing/
ready) but not a percentage, so the bar sat at 0 until the asset flipped to
ready, then snapped to 100 ("blank until it finishes").

pollRow now also fetches GET /jobs/<jobId> and feeds the BullMQ job's numeric
progress (worker emits 2..100 across the yt-dlp download + S3 upload) into the
bar, so it fills during the download. Falls back to status when the job is
evicted post-completion. Also reaffirm 'downloading' label while ingesting and
poll a bit faster (2s) since short clips finish quickly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-05-29 19:53:19 -04:00
cf1fe136d0 fix(worker): route import queue to a consumer + refresh stale yt-dlp
YouTube imports were stuck in `ingesting` forever: after the capability-
routing split (fdec2e3), every worker's WORKER_QUEUES enumerated an explicit
list that omitted `import`, so index.js's `want('import')` was false on every
worker and nothing consumed the import queue. Add `import` to worker-p4
(heavy/primary worker; import is concurrency-1 and network-bound, one consumer
is enough).

Second defect that would have surfaced once routed: the baked yt-dlp came from
the distro package (Ubuntu 22.04 apt = 2022.04.08, ~4 years stale), which
current YouTube rejects. Pull the latest self-contained yt-dlp release binary
at image build instead (GPU image: yt-dlp_linux; alpine image: python zipapp).
Rebuild the worker image to refresh.

Verified on zampp1: import jobs drain ingesting -> processing -> ready,
failed=0, yt-dlp 2026.03.17.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-05-29 19:51:27 -04:00
4 changed files with 42 additions and 5 deletions

View file

@ -116,7 +116,11 @@ services:
S3_SECRET_KEY: ${S3_SECRET_KEY}
S3_REGION: ${S3_REGION:-us-east-1}
GROWING_PATH: /growing
WORKER_QUEUES: proxy,conform,trim
# Includes `import` (YouTube importer): the import queue had no consumer
# after the capability-routing split, so import jobs sat unprocessed and
# assets stayed `ingesting` forever. import is concurrency-1 + network-
# bound, so one consumer (this heavy/primary worker) is sufficient.
WORKER_QUEUES: proxy,conform,trim,import
RUN_PROMOTION: "true"
PROXY_CONCURRENCY: "2"
NVIDIA_VISIBLE_DEVICES: GPU-79afca3e-2ab2-a6ea-1c44-706c1f0a26d6

View file

@ -230,11 +230,29 @@ function YouTubeImport({ navigate }) {
patch.error = patch.error || 'Import failed: check the Jobs screen for details.';
} else if (asset.status === 'processing') {
patch.status = 'processing';
} else if (asset.status === 'ingesting') {
patch.status = 'downloading';
}
// While the import is still running, pull the live percentage from its
// BullMQ job so the bar reflects the actual yt-dlp download instead of
// sitting at 0 until the asset flips to ready. The worker emits 2..100
// across the download + S3 upload (job.updateProgress). If the job has
// already completed and been evicted, the fetch throws and we just fall
// back to the status-driven values above.
if (row.jobId && asset.status !== 'ready' && asset.status !== 'error') {
try {
const job = await window.ZAMPP_API.fetch('/jobs/' + row.jobId);
if (typeof job.progress === 'number') patch.progress = job.progress;
} catch { /* job evicted after completion — fall back to status */ }
}
if (Object.keys(patch).length) updateRow(row.id, patch);
if (asset.status === 'ready' || asset.status === 'error') return;
} catch { /* ignore */ }
setTimeout(tick, 3000);
// Poll fairly briskly: the download phase is where the user wants to see
// the bar move, and a short clip can finish in a handful of seconds.
setTimeout(tick, 2000);
};
tick();
return () => { stopped = true; };

View file

@ -1,6 +1,12 @@
FROM node:20-alpine
# yt-dlp powers the YouTube importer; python3 is its runtime dep.
RUN apk add --no-cache ffmpeg yt-dlp python3
# Not from apk: the packaged yt-dlp goes stale and YouTube breaks old versions.
# Pull the latest python-zipapp release (runs on musl via system python3) so a
# rebuild always refreshes it. /usr/local/bin precedes /usr/bin on PATH.
RUN apk add --no-cache ffmpeg python3 curl \
&& curl -fsSL https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp \
-o /usr/local/bin/yt-dlp \
&& chmod a+rx /usr/local/bin/yt-dlp
WORKDIR /app
COPY package*.json ./
RUN npm install --omit=dev

View file

@ -9,11 +9,20 @@
FROM nvcr.io/nvidia/cuda:12.3.1-base-ubuntu22.04
# Install Node.js 20, ffmpeg (Ubuntu's ffmpeg includes h264_nvenc/hevc_nvenc),
# and yt-dlp (+ python3 runtime) for the YouTube importer.
# and yt-dlp for the YouTube importer.
#
# yt-dlp is NOT installed from apt: Ubuntu 22.04's package is pinned to a 2022
# release, which YouTube has long since broken (extraction fails). yt-dlp must
# track YouTube's frequent changes, so we pull the latest self-contained
# release binary at build time. /usr/local/bin precedes /usr/bin on PATH, so
# `yt-dlp` resolves to this one. Rebuild the worker image to refresh it.
RUN apt-get update && apt-get install -y --no-install-recommends \
curl ca-certificates ffmpeg yt-dlp python3 \
curl ca-certificates ffmpeg python3 \
&& curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \
&& apt-get install -y --no-install-recommends nodejs \
&& curl -fsSL https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp_linux \
-o /usr/local/bin/yt-dlp \
&& chmod a+rx /usr/local/bin/yt-dlp \
&& apt-get clean && rm -rf /var/lib/apt/lists/*
WORKDIR /app