How should I scale background job workers?

Separate I/O-bound jobs (email, API calls) from CPU-bound jobs (image processing, reports) into different queues. I/O workers run high concurrency (20-50 concurrent tasks) since they mostly wait for network responses. CPU workers run low concurrency (one task per core). Autoscale based on queue depth: scale up fast when the queue grows, scale down slowly when it empties.

What retry strategy should I use for failed jobs?

Use exponential backoff with jitter: delays of 1s, 2s, 4s, 8s, 16s between retries, with ±30% random jitter to prevent thundering herd effects. Distinguish transient failures (5xx, timeouts — worth retrying) from permanent failures (4xx, validation errors — send directly to dead letter queue). Limit retries to 3-5 attempts depending on priority.

Why do background jobs need to be idempotent?

In distributed systems, jobs may be delivered more than once when a worker crashes after processing but before acknowledging the job. Without idempotency, this causes duplicate emails, double charges, or duplicate data. Achieve idempotency with unique job IDs (check before processing), idempotency keys on external API calls, and database upserts instead of inserts.

How do priority queues prevent job starvation?

Create separate queues for different priorities (critical, high, default, low) and configure workers to poll higher-priority queues first using weighted fair polling: process 6 critical, 3 high, 2 default, 1 low. This ensures critical jobs like password resets are never delayed by low-priority batch operations while still making progress on all queues.

What metrics should I monitor for background jobs?

Track queue depth (are workers keeping up), job latency (enqueue to completion time, p50 and p95), processing time (per job type), failure rate (overall and per type), dead letter queue depth (unfulfilled promises), and worker utilization (optimal range 30-80%). Set alerts on rates of change, not just absolute values.

API & Backend Tom Lindgren May 26, 2026 12 min read

Background Job Performance: Processing at Scale

Background jobs are the workhorses of modern web applications. Every operation that is too slow, too unreliable, or too resource-intensive to execute within an HTTP request lifecycle gets pushed to a background job: sending emails, processing images, generating reports, syncing data with external systems, running scheduled maintenance. A typical production application processes millions of background jobs per day, and the performance of the job processing system directly affects both user experience (how quickly async operations complete) and infrastructure cost (how many workers are needed).

Designing a high-performance background job system requires balancing throughput (total jobs per unit time), latency (time from enqueue to completion), reliability (no jobs lost or duplicated), and resource efficiency (CPU and memory utilization of workers). These goals occasionally conflict — maximizing throughput through batching increases individual job latency, while guaranteeing reliability through persistent storage is slower than in-memory queues.

Job Processing Architecture

A background job system consists of three components: producers (the application code that creates jobs), a queue (the storage layer that holds jobs until they are processed), and workers (the processes that execute jobs). The performance characteristics of the entire system depend on how these components interact.

Worker Scaling Strategies

Vertical vs Horizontal Scaling

Vertical scaling (increasing per-worker concurrency) works well for I/O-bound jobs. A worker handling email sending can run 20 to 50 concurrent tasks because each task spends most of its time waiting for network responses. CPU-bound jobs (image processing, PDF generation, data analysis) benefit from horizontal scaling — more workers — because each task saturates a CPU core.

For mixed workloads, separate I/O-bound and CPU-bound jobs into different queues served by differently configured workers. I/O workers run high concurrency with modest resource allocation. CPU workers run low concurrency (one task per core) with larger memory allocation. This prevents slow CPU-bound jobs from starving fast I/O-bound jobs and optimizes resource utilization across both workload types.

Autoscaling Based on Queue Depth

Static worker counts waste resources during low traffic and create bottlenecks during spikes. Autoscaling adjusts the number of workers based on the current queue depth and processing rate. When the queue grows beyond a threshold, scale up. When the queue empties, scale down to a minimum baseline.

# Autoscaling logic (simplified)
# Scale-up thresholds
QUEUE_DEPTH_THRESHOLD = 1000     # start scaling when > 1K pending
QUEUE_AGE_THRESHOLD = 300        # or oldest job > 5 minutes
MAX_WORKERS = 20
MIN_WORKERS = 2

# Scale-down thresholds
IDLE_THRESHOLD = 60              # worker idle for > 60 seconds
SCALE_DOWN_COOLDOWN = 300        # wait 5 min between scale-downs

# Effective formula:
# desired_workers = min(
#     MAX_WORKERS,
#     max(
#         MIN_WORKERS,
#         ceil(queue_depth / target_throughput_per_worker)
#     )
# )

Scale up fast, scale down slow. When the queue grows, add workers immediately — waiting costs user-facing latency. When the queue empties, wait before removing workers — traffic often comes in bursts, and spinning up new workers has a cold-start cost (process initialization, connection establishment, cache warming).

Retry Strategies

Jobs fail. External services time out, databases hit deadlocks, resources are temporarily unavailable. A robust retry strategy distinguishes transient failures (which resolve on retry) from permanent failures (which will never succeed regardless of retry count) and handles each appropriately.

Exponential Backoff with Jitter

Retry immediately after a transient failure usually fails again — the problem (overloaded service, connection limit) has not had time to resolve. Exponential backoff increases the delay between retries geometrically: 1 second, 2 seconds, 4 seconds, 8 seconds. Adding random jitter (±30% of the delay) prevents "thundering herd" effects where many failed jobs retry simultaneously and re-create the overload that caused the original failure.

# Exponential backoff with jitter
import random

def calculate_retry_delay(attempt, base_delay=1.0, max_delay=300.0):
    """
    attempt 1: ~1s    (range: 0.7-1.3s)
    attempt 2: ~2s    (range: 1.4-2.6s)
    attempt 3: ~4s    (range: 2.8-5.2s)
    attempt 4: ~8s    (range: 5.6-10.4s)
    attempt 5: ~16s   (range: 11.2-20.8s)
    """
    delay = min(base_delay * (2 ** (attempt - 1)), max_delay)
    jitter = delay * 0.3 * (2 * random.random() - 1)
    return delay + jitter

Distinguishing Failure Types

Not all failures deserve retries. A 404 response from an external API will return 404 on every retry. A validation error means the job's input is invalid. Retrying these failures wastes worker capacity and delays processing of healthy jobs behind them. Categorize failures into retryable (5xx, timeouts, connection errors) and non-retryable (4xx, validation errors, missing data) and route non-retryable failures directly to the dead letter queue after the first attempt.

Priority Queues

Not all jobs are equally urgent. A password reset email should process within seconds. A weekly analytics report can wait minutes or hours. Without priority handling, a burst of low-priority jobs (like nightly report generation) can delay high-priority jobs (like user notifications) by exhausting worker capacity.

Implement priority queues by creating separate queues for different priority levels (critical, high, default, low) and configuring workers to poll higher-priority queues first. A common pattern is weighted fair polling: for every 6 jobs processed from the critical queue, process 3 from high, 2 from default, and 1 from low. This ensures low-priority jobs still make progress during bursts while critical jobs are never significantly delayed.

Priority	Typical Jobs	Target Latency	Retry Limit
Critical	Password resets, payment confirmations	< 10 seconds	5 (immediate retries)
High	User notifications, order processing	< 1 minute	5 (backoff retries)
Default	Email campaigns, data sync	< 15 minutes	3 (backoff retries)
Low	Reports, cleanup, analytics	< 1 hour	3 (backoff retries)

Job Batching and Bulk Processing

When many jobs perform the same type of work on different data (sending 10,000 notification emails, processing 500 uploaded images, updating 50,000 search index entries), processing them individually is wasteful. Each job independently establishes connections, initializes resources, and commits transactions — overhead that could be amortized across a batch.

Job batching groups similar jobs and processes them together. Instead of 10,000 individual "send email" jobs, create one "send email batch" job with 500 recipients. The batch job establishes one SMTP connection and sends all 500 emails over it, rather than 500 separate connection setups. Batch sizes should balance efficiency (larger batches amortize more overhead) with reliability (if a batch fails, all jobs in the batch must retry) and latency (large batches take longer to complete).

Idempotency and Exactly-Once Processing

In distributed systems, jobs may be delivered more than once. A worker crashes after processing a job but before acknowledging it. The queue redelivers the job to another worker. If the job sends an email, the user receives two identical emails. If the job charges a payment, the customer is charged twice.

Design every job to be idempotent: processing the same job multiple times produces the same result as processing it once. Achieve idempotency through:

Unique job IDs: Assign each job a UUID at creation time. Before processing, check if a job with that ID has already been completed. If yes, skip it.
Idempotency keys on external calls: Pass a unique key with each external API call (payment processors, email services). The external service deduplicates requests with the same key.
Database upserts: Use INSERT ON CONFLICT (upsert) instead of INSERT for database writes. If the job runs twice, the second execution updates the existing row rather than creating a duplicate.

Monitoring Background Job Systems

Background job systems fail silently. Unlike API endpoints where users immediately notice errors (timeout pages, error responses), a failed background job produces no user-visible symptom until someone notices the email never arrived, the report is missing, or the data sync is hours behind. Proactive monitoring is essential.

Key Metrics

Track these metrics across all queues and job types in your APM system:

Queue depth: Current number of pending jobs. Growing depth indicates workers are not keeping up with production rate. Set alerts on sustained growth.
Job latency: Time from enqueue to completion (includes queue wait time plus processing time). Track p50 and p95 separately — the p95 reveals problems that the median hides.
Processing time: Time spent actually executing the job (excludes queue wait time). Use this to identify slow job types and optimize their implementation.
Failure rate: Percentage of jobs that fail on first attempt. A spike in failure rate across all job types suggests an infrastructure problem. A spike in one job type suggests a code or data issue.
Dead letter queue depth: Jobs that exhausted all retries. A growing DLQ always warrants investigation — each entry represents a user-facing promise that was not fulfilled.
Worker utilization: Percentage of worker capacity in use. Below 30 percent means over-provisioned (reduce workers). Above 80 percent means under-provisioned (add workers before queue depth grows).

Set up alerts not only on current metric values but on rates of change. A queue depth of 10,000 is normal during a batch import but alarming if it was 100 five minutes ago. Rate-of-change alerts catch problems earlier than static thresholds because they detect the acceleration before the absolute value becomes critical.

Job-Level Tracing

Integrate background jobs with your distributed tracing system. Propagate the trace context from the HTTP request that created the job through the queue and into the worker. This allows you to see the complete lifecycle of a user action: the API request that enqueued the job, how long the job waited in the queue, which worker processed it, what external calls the worker made, and when the job completed. Without tracing, debugging "why did the user not receive their confirmation email" requires searching through logs across multiple services.