HLS Video Pipeline

Production-ready video transcoding with FFmpeg and BullMQ.

Source

Overview

A video processing pipeline that takes uploaded source files and produces HLS streams ready for adaptive playback. The goal was a small, reliable workflow that could run in production without a heavyweight media stack — upload a file, track its status, and play it back from a simple web UI.

I did not want a black-box transcoder. I wanted something I could reason about: a queue, workers, FFmpeg jobs, manifests on object storage, and a library UI that shows what is ready to watch.

The problem

Uploading a large video and waiting for “processing…” is fine until a job fails halfway, two uploads compete for CPU, or you need more than one bitrate for mobile. HLS (HTTP Live Streaming) solves the playback side by packaging multiple renditions and a playlist. The hard part is the middle: reliable jobs, retries, and storage paths players can actually fetch.

What I built

  • Upload ingestion with validation and job enqueueing
  • FFmpeg-based transcoding into multiple HLS renditions
  • BullMQ workers for retries, concurrency control, and failure handling
  • Job status tracking so the UI can show queued, processing, ready, or failed
  • A playback UI with a video library sidebar and search
  • Storage and playback paths wired for streaming clients

Pipeline flow

  1. Upload — the source file is accepted and stored.
  2. Enqueue — a BullMQ job captures the file id and desired renditions.
  3. Transcode — an FFmpeg worker produces segmented streams and an HLS playlist.
  4. Publish — segments and manifests land in object storage.
  5. Play — the web UI loads the playlist and streams adaptively.

Stack

Node.js, FFmpeg, BullMQ, Redis, and object storage for segments and manifests. The worker layer is intentionally separate from the upload API so CPU-heavy jobs do not block the web process.

What I learned

Video jobs fail in boring ways: missing codecs, bad containers, disk pressure, and timeouts. Retries help only if failures are classified. Treating “bad input” differently from “worker died” keeps the queue honest and the UI useful.

What's next

Webhook callbacks on job completion, thumbnail generation, and tighter observability around queue lag so you can see when workers are falling behind before users do.