Reliable background job processing with Spring Boot and Redis

Implementing a job queue for background task processing with Spring Boot and Redis lets web requests return quickly while invoices, payment updates and notifications run asynchronously. Redis provides a shared buffer that absorbs traffic bursts and scales across application instances.

Australian merchants in Sydney, Melbourne, Brisbane and Perth often serve customers beyond standard office hours, so receipt emails, delivery updates and bank reconciliation must survive uneven mobile coverage and sleeping devices. Systems handling Australian dollars, GST records or personal information must also respect the Privacy Act 1988 (Cth) and the Spam Act 2003 (Cth) when personal data or commercial messages are involved.

Choosing the right Redis primitive

Redis lists are attractive because producers can use LPUSH while workers use BLMOVE. They suit uniform jobs, but retries, ownership and failure history need extra structures. An unacknowledged message also makes recovery harder to observe.

Redis Streams adds ordered identifiers, consumer groups and a pending-entry list. These features suit mixed task types and expose stalled work. For most Spring Boot services, Streams is the stronger default; choose a list only when basic first-in, first-out delivery is sufficient.

Modelling background work

Represent work as a versioned message rather than a method name. Include a unique job ID, an idempotency key and the minimal payload needed by the consumer.

public record JobMessage(
    String jobId,
    String type,
    String payload,
    String idempotencyKey,
    Instant createdAt) {}

Store payloads as JSON and keep domain data in a database where practical. This makes schema evolution explicit, reduces accidental personal-data retention in Redis and keeps queued messages portable between workers.

Publishing jobs without losing work

Create the workers consumer group at startup and tolerate the BUSYGROUP response when it already exists. Publishing is then a single XADD operation:

redisTemplate.opsForStream().add("jobs", Map.of(
    "jobId", job.jobId(),
    "type", job.type(),
    "payload", job.payload(),
    "idempotencyKey", job.idempotencyKey()));

When the job originates in a database transaction, avoid an unsafe database-then-Redis dual write. Save an outbox row in the same transaction, then let a scheduled publisher move it to Redis and mark it sent. Reuse the outbox ID as the idempotency key to cover a publisher restart between those steps.

Consuming tasks with safe delivery

Spring Data Redis can read a bounded batch, block briefly for new entries and acknowledge each record only after handling succeeds:

StreamOperations<String, String, String> stream = redisTemplate.opsForStream();
List<MapRecord<String, String, String>> records = stream.read(
    Consumer.from("workers", instanceId),
    StreamReadOptions.empty().count(10).block(Duration.ofSeconds(2)),
    StreamOffset.create("jobs", ReadOffset.lastConsumed()));

Execute the handler, persist its result and call stream.acknowledge("jobs", "workers", record.getId()). Do not acknowledge before the side effect finishes. If a process dies between the side effect and acknowledgement, Redis may deliver the message again, so the idempotency key must protect against duplicate invoices, notifications or payments.

Retrying failures and recovering pending work

Treat connection resets and temporary downstream failures as transient, while validation errors and rejected records are permanent. Retry transient work a few times with exponential backoff, then move it to a dead-letter stream with the original payload, error and attempt count.

A crashed worker leaves entries in the stream’s pending list. A recovery task can use XAUTOCLAIM to assign stale entries to healthy workers after a configurable idle period. Store attempt counts separately in a Redis hash, and never block a worker thread with Thread.sleep; schedule the retry instead.

Securing Redis and Australian data

Run Redis privately with TLS, network rules, least-privilege credentials and encrypted backups. Choose an Australian region such as ap-southeast-2 where available. When selecting infrastructure, teams can review managed hosting options alongside the Redis provider’s regional plan. Keep application logs free of raw payloads.

The Privacy Act 1988 governs many Australian organisations, while commercial messages may engage the Spam Act 2003; state privacy rules can add further duties. Define payload retention, deletion and access controls before storing customer records, and rotate credentials without restarting every worker.

Queue design and production checklist

Agree on the queue’s delivery guarantees and failure behaviour before deployment:

Core queue decisions

• Choose Redis Streams and stable consumer-group names.
• Store versioned JSON with job and idempotency keys.
• Acknowledge only after successful side effects.
• Set retry limits and a dead-letter stream.

Operational safeguards

• Keep Redis private, encrypted and near the Boot region.
• Track queue depth, pending age and failure rate.
• Test crashes, Redis failover and database outages.
• Document replay, retention and privacy controls.

For a tailored architecture review or production rollout, JavaWhizz pairs focused Java guidance with IT consultancy services. The result is a queue that keeps request threads responsive while giving operations clear signals for retries, isolation and recovery.