Designing a Reliable Job Queue in WordPress

/ Definition

What is a job queue in WordPress?

A job queue is a persistent list of work to be done later, drained by a background worker. Instead of running an expensive task inside the request that triggered it, you write a small row describing the task — the job — and return immediately. Something else picks that row up moments later and does the actual work. In WordPress that "something else" is almost always driven by a WP-Cron tick, and the list itself is a database table.

The value is in decoupling. The user who placed the order, or the admin who clicked "sync 5,000 products", gets an instant response; the heavy lifting happens out of band, at a pace the server can sustain. This is the concrete, table-level view of the broader idea covered in WordPress async background processing — here we focus on the queue itself.

/ Why not WP-Cron

Why not just use WP-Cron for background jobs?

WP-Cron is a scheduler, not a queue. It stores every event in a single serialised cron array inside wp_options, runs on page loads, keeps no history, and never retries a failure. That is perfectly good for "run this once tonight." It falls apart when you have thousands of discrete jobs that each need to be claimed exactly once, retried on error, and audited afterward — there is nowhere in a serialised option to track per-job status or attempt counts.

/ Schema

What does a job queue look like in the database?

At its simplest, a queue is one table. Each row carries the payload, a status, an attempt count, and timestamps. The status and available_at columns are what the worker filters on; the attempts column is what turns a transient failure into a retry rather than a lost job.

/ Claiming

How does a worker claim and process jobs safely?

The hard part of any queue is making sure two concurrent workers never run the same job. The race-safe technique is a single conditional UPDATE: flip a batch of pending rows to processing and stamp them with a worker/claim marker, filtering on the current status so only one worker can win each row. Whatever rows that UPDATE actually changed are yours to process.

PHP — claim a batch, then process it

global $wpdb;
$claim = wp_generate_uuid4();

// Atomically reserve up to 10 due jobs for THIS worker
$wpdb->query( $wpdb->prepare(
    "UPDATE {$wpdb->prefix}my_jobs
     SET status = 'processing', claim = %s, reserved_at = %s
     WHERE status = 'pending' AND available_at <= %s
     ORDER BY id LIMIT 10",
    $claim, current_time( 'mysql' ), current_time( 'mysql' )
) );

// Only rows carrying our claim are ours to run
$jobs = $wpdb->get_results( $wpdb->prepare(
    "SELECT * FROM {$wpdb->prefix}my_jobs WHERE claim = %s", $claim
) );

This is exactly the pattern Action Scheduler formalises with its separate claims table — the claim marker is what guarantees single execution under concurrency.

A job queue is correct when a job is claimed by exactly one worker and either completes or is retried — never silently dropped, never run twice. — the core queue invariant

/ Retries

How should a queue handle retries and failures?

Wrap each job so a thrown exception does not kill the whole batch. On failure, increment attempts, and if it is still under the limit push the job back to pending with a future available_at — this is where exponential backoff belongs, spacing retries out as failures accumulate. Once attempts hits the ceiling, mark the row failed so it stops consuming worker time and shows up for a human to inspect. That "give up cleanly and keep the evidence" step is what separates a real queue from an infinite retry loop.

1. Run inside a try/catch. One bad job must not abort the other nine in the batch.
2. On error, increment attempts. If under the limit, reset status to pending and push available_at into the future (backoff).
3. At the limit, mark failed. Stop retrying, keep the row and its last error for inspection.

/ Libraries

What libraries give you a job queue in WordPress?

You rarely need to hand-roll all of this. Three common options cover most needs, and each is a variation on the same enqueue-claim-process loop.

For the overwhelming majority of cases the answer is Action Scheduler — it already implements claiming, retries, and an audit trail correctly. Its internals are a useful reference for how a mature queue is structured; see the Action Scheduler database schema and the PHP API reference.

/ When

When do you need a queue vs a single scheduled event?

Reach for a queue when work is high-volume, must retry on failure, needs a per-item trail, or must not block the request. A single delayed task — one reminder email, one cleanup pass — is fine on a lone scheduled event and does not justify a table and a worker. The moment you have many independent jobs arriving in bursts, each of which can fail on its own, you want the queue.