Node.js Redis and BullMQ Tutorial

What and Why

Some work has no business happening inside an HTTP request: sending emails, generating PDFs, calling slow third-party APIs, resizing uploads. If you do it inline, your users wait, and a failure tears down the whole response. The fix is a background job queue.

BullMQ is a Node.js job queue built on Redis. It stores jobs in Redis lists and sorted sets, hands them out to worker processes, and handles retries, delays, priorities, and concurrency. You get durability, observability, and horizontal scaling without writing your own state machine.

Mental Model

There are three roles. Producers add jobs to a queue. Workers pull jobs and execute them. Redis is the shared state where jobs live between the two. Producers and workers do not know about each other; they only know the queue name.

Each job has a name, a payload, and options like attempts, backoff, and delay. BullMQ guarantees that a job is delivered to exactly one worker at a time, retries on failure, and surfaces final failures into a dead-letter-style failed set you can inspect.

Hands-on Example

A queue that sends welcome emails with retries.

// queue.js
import { Queue } from 'bullmq';
export const emailQueue = new Queue('emails', {
  connection: { host: '127.0.0.1', port: 6379 },
});

// producer.js
import { emailQueue } from './queue.js';
await emailQueue.add(
  'welcome',
  { userId: 42, to: 'a@example.com' },
  { attempts: 5, backoff: { type: 'exponential', delay: 1000 } },
);

// worker.js
import { Worker } from 'bullmq';
new Worker(
  'emails',
  async (job) => {
    if (job.name === 'welcome') {
      await sendWelcome(job.data);
    }
  },
  { connection: { host: '127.0.0.1', port: 6379 }, concurrency: 10 },
);

Run the worker in its own process. Crash it, restart it, and the queue picks up where it left off because Redis is the source of truth.

Producer --add--> [ Queue in Redis ]
                      |
      +---------------+----------------+
      |               |                |
      v               v                v
   Worker A        Worker B         Worker C
      |
   success / fail -> Redis (completed / failed sets)

Common Pitfalls

Sharing one Redis connection across producers and workers without maxRetriesPerRequest: null causes BullMQ to throw on reconnects. Read the BullMQ Redis options once and apply them everywhere.

Doing slow CPU work inside a worker blocks its event loop and starves other jobs. Either lower the worker concurrency or move CPU work into worker threads.

Forgetting idempotency leads to duplicate side effects when a job retries. Always design job handlers so running them twice with the same payload is safe, often by keying off an idempotencyKey you store in your database.

Not setting removeOnComplete and removeOnFail leaves Redis growing forever. Cap completed and failed history to something reasonable like a few thousand entries.

Practical Tips

Use one queue per job family rather than one big queue with branching logic. Smaller queues are easier to monitor and scale independently.

Set sensible defaults on the queue: attempts, backoff, removeOnComplete. Producers should override only when they really need to.

Add a separate worker process per queue type. That way a slow PDF worker cannot starve fast email workers, and you can scale each pool independently.

Use BullMQ’s QueueEvents to emit metrics on completed, failed, and stalled jobs. Pipe those to Prometheus or your dashboard so you see queue depth and failure rates at a glance.

For recurring jobs, use repeat with a cron expression. Avoid running cron in your application; the queue is the schedule of record.

Wrap-up

Redis plus BullMQ turns ad hoc background work into a real system with retries, scheduling, and visibility. Keep producers thin, workers idempotent, queues focused, and Redis well-tuned. With those pieces in place, you can move almost any slow work out of the request path and into a pipeline you can actually reason about.