Queues and asynchronous work

EverDuet runs BullMQ on a private Redis service, but intentionally has only one job: system.probe. This is not unfinished feature work. It is a deliberately exercised boundary waiting for a measured use case.

What a queue actually buys

A message queue separates accepting work from completing it. That can provide:

• bounded concurrency;
• retries with backoff;
• delayed or scheduled execution;
• independent worker scaling;
• resilience when an optional dependency is briefly unavailable.

It also introduces eventual consistency, duplicate delivery, operational state, and a second place to diagnose failures.

Why BullMQ here?

BullMQ fits the existing TypeScript/Bun/Node stack, has strong Redis-backed retry primitives, and requires less operational machinery than RabbitMQ for this project's current learning goal. Redis was already the chosen queue substrate; BullMQ adds typed job behavior rather than forcing the app to invent lists, leases, acknowledgements, and retry timing.

The typed contract

Job names, payloads, and results are one discriminated mapping:

export const jobNames = {
  systemProbe: 'system.probe',
} as const

export type QueueJobPayloads = {
  'system.probe': { requestedAt: string }
}

export type QueueJobResults = {
  'system.probe': { processedAt: string }
}

The dispatcher rejects malformed and unknown work. A new job must add a payload, result, handler, and tests instead of passing untyped JSON through the system.

Different Redis behavior by role

Producers are part of request paths. They should fail promptly rather than hanging indefinitely:

const producerOptions = {
  lazyConnect: true,
  maxRetriesPerRequest: 1,
}

Workers are long-running consumers. BullMQ requires unlimited command retries so they can reconnect after Redis restarts:

const workerOptions = {
  lazyConnect: true,
  maxRetriesPerRequest: null,
}

Same infrastructure, different failure expectations.

Retry policy is part of product semantics

const defaultJobOptions = {
  attempts: 3,
  backoff: { delay: 1_000, type: 'exponential' },
  removeOnComplete: { age: 86_400, count: 1_000 },
  removeOnFail: { age: 604_800, count: 5_000 },
}

Retries imply duplicate execution. A future handler must be idempotent, usually by carrying an identifier and claiming or upserting durable state in PostgreSQL.

WARNING

Never put private prompt-answer content in a job payload. Put a safe identifier in the job and load authorized data at execution time.

Redis is bounded on purpose

Production Redis has a 96 MB data ceiling inside a 192 MB container and uses noeviction. At capacity, new queue writes fail visibly instead of silently evicting BullMQ state. Completed and failed job retention is also bounded.

This is a useful example of choosing correctness over apparent availability: a visible enqueue failure is diagnosable; silently losing queue bookkeeping is not.

Why not use the queue yet?

Current user mutations are fast and benefit from immediate, explicit success or failure. Moving them to BullMQ would add eventual consistency without measured latency value.

A good first product job has all of these properties:

1. optional to the primary transaction;
2. slow or dependent on a flaky external service;
3. safe to retry;
4. representable by identifiers rather than private content;
5. observable when permanently failed.

Potential examples are future notification delivery or batch generation follow-up—not relationship membership or response ownership.

The canary proves the whole seam

bun run queue:probe:

1. connects as a producer;
2. registers for queue events;
3. adds a uniquely identified probe;
4. waits for the worker result;
5. closes every resource on success or timeout.

The Coolify PR gate runs the same canary against ephemeral Redis. This verifies more than “Redis answered PING”; it proves producer, event stream, dispatcher, worker, and result flow.

Next: testing and delivery.