Chapters of From Frontend to Full-Stack Backend

Week 3 - Interview-এ সবচেয়ে বেশি জিজ্ঞেস করা TopicsChapter 16 of 283 min read

Queues - pushing slow work to the background with BullMQ

Why not all work belongs inside the request, background jobs, retries and failed job handling

After an order is confirmed, the customer has to be sent an email and an SMS. The email server was slow that day - it took 4 seconds. Your API also stayed stuck for 4 seconds, and the customer, watching the spinning wheel and thinking the order never went through, pressed the button again - two orders! The real problem is here: work whose result does not have to be given to the user at once should never have been kept inside the request in the first place.

The queue idea - putting work in a line

A queue is a line of work. The API only puts the job in the line ('send this order's email') and gives the response at once. Behind the scenes, a separate worker picks jobs from that line one by one and does them. The user's experience becomes snappy, and the slow jobs keep happening behind at their own pace. BullMQ does exactly this job - it uses Redis underneath, which was already set up in the previous chapter.

import { Queue, Worker } from "bullmq";

const emailQueue = new Queue("email", { connection: redis });

// Inside the API - only putting it in line, not sending
await emailQueue.add("order-confirmation", { orderId, email });
res.status(201).json({ success: true, data: order });

// A separate worker - the real sending work is here
new Worker("email", async (job) => {
  await sendOrderEmail(job.data.orderId, job.data.email);
}, { connection: redis });

Retry - a failure is not giving up

The email server may be temporarily down. The beauty of a queue is that a failed job can be tried again - like 3 times, waiting a little longer each time (backoff). If it still fails, the job stays in a failed state, so it can be looked at later and fixed by hand or run again. No job quietly disappears - that is the big advantage over doing it inside the request.

The rule for knowing which work goes to a queue: if the work is slow, depends on an outside service (email, SMS, payment gateway), or you want another chance if it fails - put it in the queue. Work without whose answer you cannot give a response at all (like a login check) - that stays inside the request.

The mistake everyone makes

The most seen mistake: doing slow work like sending emails/SMS, processing images or building reports inside the controller. That makes the API slow, increases timeouts, and when the outside service goes down your whole API gets stuck.

Today's practice

Move the confirmation-email job after order creation into a BullMQ queue (for now just logging to the console instead of sending a real email is fine). Deliberately cause an error once in the worker and check whether a retry happens.

How to say it in an interview

I send slow or outside-service-dependent work to a queue - the API only adds the job and gives the response at once, the worker does the job behind. BullMQ has retries and backoff, so temporary failures do not lose work; if it finally fails, it stays as a failed job that can be looked at later.

All chapters of From Frontend to Full-Stack Backend