Backend Execution Model
This is the mental model for how Threadmill moves a job through Postgres and Redis under parallel workers. It explains where the atomic boundary is, which state transitions are visible to other nodes, and where concurrency and starvation protections live.
Threadmill gives at-least-once delivery. A job can run more than once after a crash, timeout, orphan reclaim, or retry. The execution model prevents double-claim and same-key overlap; it does not make handler side effects exactly once.
Job State Shape
Section titled “Job State Shape”Only ENQUEUED jobs are claimed by a dispatcher. SCHEDULED jobs must first
be promoted by maintenance, and AWAITING workflow jobs must first be promoted
by WorkflowInterceptor.
For per-key concurrency, Threadmill treats ENQUEUED, SCHEDULED, and
AWAITING as pending. That matters because a later reader must not leapfrog an
earlier writer just because the writer is still scheduled or waiting on a
workflow predecessor.
Within one concurrency key, pending order is total: first by current_state_at,
then by JobId. Priority chooses which queue candidates are inspected first,
but it does not let later same-key work bypass earlier same-key work.
stateDiagram-v2 [*] --> SCHEDULED: delayed enqueue [*] --> ENQUEUED: immediate enqueue [*] --> AWAITING: workflow child
SCHEDULED --> ENQUEUED: maintenance promotion AWAITING --> ENQUEUED: parent succeeded ENQUEUED --> PROCESSING: atomic claim
PROCESSING --> SUCCEEDED: handler returned PROCESSING --> FAILED: exception, timeout, no progress, orphan reclaim PROCESSING --> QUARANTINED: payload or handler cannot be loaded PROCESSING --> DELETED: explicit delete AWAITING --> DELETED: abandoned workflow descendant
FAILED --> SCHEDULED: retry interceptor schedules another attemptThe handler runs only while the job is PROCESSING. Threadmill commits the
claim before calling user code, and opens a separate save operation after user
code returns or fails.
Per-Key Claim Rule
Section titled “Per-Key Claim Rule”For a job with concurrencyKey = K:
SHAREDcan claim when noEXCLUSIVEjob is currentlyPROCESSINGforKand no earlier pendingEXCLUSIVEjob exists forK.EXCLUSIVEcan claim when no job is currentlyPROCESSINGforKand no earlier pending job exists forK.
The second rule is deliberately strict for EXCLUSIVE: it keeps in-key order
stable and avoids a writer jumping ahead of earlier pending work for the same
resource.
Workflow children inherit the root job’s key and mode. A workflow root owns one
hold for the whole workflow, and the hold is released only when the last
descendant reaches a release state: SUCCEEDED, FAILED, DELETED, or
QUARANTINED. A retry is a later, separate FAILED -> SCHEDULED transition;
that next attempt must acquire the key again.
Postgres Path
Section titled “Postgres Path”Postgres uses normal row-level database locking. The queue decision and the state write happen in one JDBC transaction; the handler never runs inside that transaction.
sequenceDiagram participant D as Dispatcher participant PG as PostgresJobStore participant J as threadmill_jobs participant G as threadmill_concurrency_groups participant H as Handler
D->>PG: claimReady(queue, capacity) PG->>J: page scalar candidates FOR UPDATE SKIP LOCKED PG->>G: lock distinct group rows once, sorted by key PG->>J: batch-load active holds and earliest pending order loop each claimable candidate PG->>J: fetch body PG->>J: UPDATE state to PROCESSING, owner, attempts, version alt first job in workflow hold PG->>G: increment shared or exclusive in-flight counter end end PG-->>D: claimed jobs after COMMIT D->>H: run job outside Threadmill transaction H-->>D: success or failure D->>PG: saveAtomic terminal transition PG->>J: SELECT current job row FOR UPDATE PG->>G: lock group row for old concurrency key PG->>J: UPDATE terminal state, version, body PG->>G: decrement counter if workflow hold is completePostgres Parallelism
Section titled “Postgres Parallelism”- Different queues can be claimed by different dispatchers and lanes.
- Different candidates in the same queue are protected by
FOR UPDATE SKIP LOCKED, so competing nodes skip rows already being claimed. - Candidate gathering uses bounded unkeyed, keyed-head, and active-hold lanes.
The keyed lane examines at most 512 queue-scoped keys per unproductive poll
and rotates a process-local cursor so later keys are reached without walking
the pending-row backlog. Each
PostgresJobStoreinstance rotates independently; cursor entries are bounded and disposable fairness hints, not durable scheduling state. When the bound is exceeded, least-recently-used eviction preserves active queues’ cursors and discards an idle queue’s hint. Eviction preserves claim correctness but restarts that queue’s fairness progress. - Claim decisions are scalar-first: group counters, active workflow holds, and earliest pending order are loaded once per page, and job bodies are fetched only for candidates that will actually claim.
- Different concurrency keys mostly proceed independently.
- The same concurrency key serializes through the group row lock while a claim or release decision is made.
- User handlers do not hold queue row locks, so long-running jobs do not block unrelated claims.
Postgres can still build a backlog when offered more work than the local database, connection pool, worker count, or queue topology can drain. A growing queue with passing concurrency invariants is a capacity signal, not by itself a same-key correctness failure.
Oracle Path
Section titled “Oracle Path”Oracle follows the Postgres path — the same tables, row locks, group rows, and
workflow holds in one transaction — with three mechanical differences. Oracle
rejects a row limit on a locking query, so locking cursors are index-ordered and
fetched only as far as the page budget; Oracle locks SKIP LOCKED rows as they
are fetched. Keyed candidates are first read through one top-n branch per key
(at most 128 keys per unproductive poll, rotated like Postgres) and then locked
by primary key with SKIP LOCKED, dropping rows another claimer holds. The
leapfrog rule is decided by EXISTS range probes bounded at each candidate’s own
current_state_at. See the
Oracle store README.
Redis Path
Section titled “Redis Path”Redis uses server-side Lua scripts as the atomic boundary. Java may inspect
candidate hashes before the claim, but the final decision is made inside
claim_commit.lua.
sequenceDiagram participant D as Dispatcher participant R as RedisJobStore participant Q as queue ZSET participant L as key claim lock participant S as claim_commit.lua participant C as counters and concurrency indexes participant H as Handler
D->>R: claimReady(queue, capacity) R->>Q: gather candidates from key-driven lanes loop each candidate R->>R: read job hash and prepare PROCESSING body alt candidate has concurrency key R->>L: acquire short per-key claim lock end R->>S: verify state, version, queue membership S->>C: inspect counters, pending members, workflow count S->>C: increment workflow/counter hold and remove pending member S->>Q: remove from ENQUEUED queue index S-->>R: OK or BLOCKED alt candidate has concurrency key R->>L: release claim lock end end R-->>D: claimed jobs D->>H: run job outside Redis script H-->>D: success or failure D->>R: saveAtomic terminal transition R->>R: save_atomic.lua updates body, indexes, counts R->>C: decrement workflow/counter hold if completeThe short per-key claim lock is not the logical concurrency hold. It only
serializes the claim decision for one key while Java prepares and submits the
Lua commit. The durable scheduling state is the job hash plus the per-key
counter, pending, workflow-count, and active workflow-hold structures. If a
client races or the lock expires, the Lua script still verifies the job
version, current state, queue membership, and key counters before it can move
anything to PROCESSING.
All Redis engine keys live under the {threadmill} slot, so the scripts update
the job body, active-state indexes, counts, concurrency counters, pending
members, workflow counts, and workflow holds atomically on one Redis Cluster
slot.
Redis Parallelism
Section titled “Redis Parallelism”- Redis executes one script at a time, so each claim or save is internally serial.
- Multiple Java dispatchers can scan candidates concurrently, but only a Lua script can commit a claim.
- Candidate gathering is key-driven, never backlog-walking: unkeyed heads come from a per-queue unkeyed ZSET, keyed candidates come from each key’s queue-specific ready index (keys are discovered through a bounded ordered registry page). Rotating windows reach active-workflow-hold members behind blocked heads; admission probes only the earliest pending barrier. A blocked hot key costs a handful of reads per pass regardless of how many jobs are queued behind it.
- Different concurrency keys avoid logical interference, although they still share one Redis server execution lane.
- The same concurrency key is additionally serialized by the per-key claim lock while claim preparation is in progress.
- Workflow outstanding counts are maintained incrementally in a separate hash; the active workflow-hold hash only means “this workflow currently owns the key.”
- Handlers run outside Redis scripts, so a slow handler does not block Redis from serving claims or saves.
Redis usually has lower local claim latency than Postgres because the claim decision is a small in-memory script instead of a SQL transaction with row locks, indexes, and connection-pool scheduling. That is a performance profile, not a different correctness contract.
Bookkeeping Paths
Section titled “Bookkeeping Paths”The diagrams show the hot claim and save path. The same concurrency bookkeeping also applies to the quieter mutation paths:
insert,insertAll, andenqueueIfAbsentadd keyed pending jobs to the pending structure when their state isENQUEUED,SCHEDULED, orAWAITING. Redis keyed inserts take the short per-key claim lock before updating workflow bookkeeping, so they cannot race a same-key claim that is preparing the first workflow hold. Redis also increments the separate workflow-count hash for keyed non-terminal jobs.- Workflow child insertion inherits the parent’s
workflow_root_id,concurrencyKey, andConcurrencyMode. If the root hold is already active, the workflow outstanding count is incremented. replaceJobmoves a pending job between old and new pending structures when the replacement changes the queue, scheduled time, key, or mode.saveAtomicremoves old pending membership, adds new pending membership when a retry or workflow promotion creates pending work, and releases active holds on release-state transitions. Redis decrements workflow counts when a keyed job becomes terminal and increments them when a terminal job re-enters a pending state for retry.softDeleteremoves active or pending indexes and releases the hold when aPROCESSINGjob is deleted.
Starvation And Fairness
Section titled “Starvation And Fairness”Threadmill handles the starvation cases that would break correctness:
- A stream of
SHAREDjobs cannot starve an earlier pendingEXCLUSIVEjob for the same key. - An
EXCLUSIVEjob cannot run beside any other job for the same key. - Workflow descendants cannot release the key early; the root hold survives until the workflow is terminal.
- Orphan reclaim, timeouts, thrown exceptions, and quarantine all release through the same persisted transition path.
- Queue-family stride scheduling smooths dispatch across matched queues inside one lane, and a zero weight pauses a queue in that family.
- PostgreSQL rotates through concurrency keys in pages of 512. A claimable key
is inspected within approximately
ceil(distinct keys / 512) × pollIntervalwhen earlier pages remain concurrency-blocked. An unproductive claim stops after one page so its transaction does not grow theFOR UPDATElock set with key count. Wake signals remain hints: above one page they wake the current cursor page, which may not contain the newly enqueued key.
Threadmill does not promise:
- Global FIFO across different queues.
- A latency bound when offered load is above backend or worker capacity.
- Per-queue rate limiting.
- Exactly-once handler side effects.
- Fairness between unrelated applications sharing the same datastore.
Race Checklist
Section titled “Race Checklist”Use this checklist when reasoning about a production incident or a soak result:
- Did two nodes claim the same job? The expected protection is job version, state, and queue membership verification inside the atomic claim boundary.
- Did two incompatible jobs for the same key run together? The expected protection is the Postgres group row or Redis per-key counter and workflow hold.
- Did a writer wait behind later readers? The expected protection is the
earlier-pending check for
SHAREDcandidates. - Did a workflow release the key before its children ended? The expected protection is the workflow-root outstanding count.
- Did the queue grow while invariants held? Treat that first as capacity, pool sizing, lane topology, or workload-shape pressure.
- Did a key stay blocked after failure? Check that the job left
PROCESSINGthrough the normal save path and that descendants were promoted or abandoned as expected.