threadmill-store-redis
Redis backend for the JobStore SPI. Per-job HASH plus auxiliary indexes
kept in sync via atomic Lua scripts. Standalone, Sentinel, and Cluster
topologies are all supported.
Topology
Section titled “Topology”Choose via RedisStoreConfig:
- Standalone — single Redis instance, optionally with a password.
- Sentinel — master/replica with automatic failover via Sentinels. Redis data-node and Sentinel ACL credentials are configured independently; one TLS policy covers both connection planes, as required by Lettuce’s Sentinel URI.
- Cluster — multi-node Cluster; every engine key uses the single
{threadmill}hash tag so Lua scripts run inside one slot. This supports Cluster topology/failover; it deliberately does not shard job keys across masters. ACL authentication and verified TLS apply to every seed.
For example, an authenticated TLS Cluster can be configured directly:
var config = new RedisStoreConfig.Cluster( List.of(new RedisStoreConfig.HostAndPort("redis-1", 6380)), "master", new RedisStoreConfig.Credentials("threadmill", clusterPassword), RedisStoreConfig.Tls.verified());var store = new RedisJobStore(config);Credentials also supports password-only authentication. Peer verification is
enabled by Tls.verified() and uses the JVM trust material. For custom Lettuce
TLS resources or client options, inject a caller-managed RedisClient or
RedisClusterClient; closing the store does not close an injected client.
Threadmill topology descriptions and wrapped connection failures never include
ACL usernames or passwords.
Redis 7.4 or later is required on every data node, including replicas that
may be promoted. Startup checks the connected server with INFO server against this tested
support baseline. Keep all nodes in a topology on supported
versions. Managed services that restrict INFO/CONFIG may use
RedisSafetyValidation.externallyValidatedMode() only after independently
verifying both the version and the no-eviction policy. Tests and examples use
the minimum supported 7.4 release line.
This is a qualification boundary, not a claim that each script needs a 7.4-only
command. Redis 7.0/7.2 and Valkey have not passed Threadmill’s supported
version/topology matrix and are not supported. externallyValidatedMode() is
for restricted administrative commands on an otherwise supported deployment;
it does not extend the supported version or product matrix.
RedisJobStore implements AutoCloseable: use try-with-resources for manually
owned stores. Closing a store closes its connection; a caller-injected client
remains owned by the caller.
Durability is Redis-level. Run with --appendonly yes. Out of the box,
Redis is less durable than PostgreSQL — a crash within 1 s of a state
change may lose the state change depending on appendfsync setting.
Crash-mid-claim semantics are still correct (orphan recovery runs), the
durability question is whether the engine remembers what it did.
The published module uses Lettuce 6.8.2.RELEASE and forces its Netty graph to
the 4.1.137.Final security floor. Lettuce 6.8 promotes DNS resolution to a
compile dependency, so netty-resolver-dns and netty-codec-dns are expected
runtime dependencies of this module; do not exclude them as accidental
transitives. See Dependency security for the
advisory-specific reachability record.
Threadmill also requires maxmemory-policy noeviction. Eviction policies are
cache semantics, not durable job-store semantics: allkeys-* can split job
hashes from their indexes/counts, and volatile-* can delete TTL-backed
heartbeats, leases, mutexes, or claim locks. RedisJobStore validates the
policy on startup unless RedisSafetyValidation.externallyValidatedMode() is
used for managed Redis where CONFIG GET is unavailable.
Key layout
Section titled “Key layout”Every key lives under {threadmill}: so multi-key Lua scripts route to
the same Cluster slot. The userSegment encoding is Base64Url(value) so
queue / handler / dedup-key user input cannot escape the namespace. Optional
Lua key positions use {threadmill}:no_key, never an empty key, so absent
indexes remain in the same slot too.
| Key | Type | Purpose |
|---|---|---|
{threadmill}:job:{id} |
HASH | Per-job state. Fields: body, state, queue, priority, handler_signature, scheduled_at, owner_node_id, owner_heartbeat_at, last_checkin_at, current_state_at, created_at, workflow_root_id, concurrency_key, concurrency_mode, version, execution_revision. |
{threadmill}:queue:{queue} |
ZSET | ENQUEUED job ids per queue. Score -priority so ZRANGE LIMIT 0 N returns highest-priority first; Redis breaks equal-score ties lexicographically by the UUIDv7 job-id member. This exactly matches (priority DESC, id) across the full int priority and timestamp ranges. |
{threadmill}:scheduled |
ZSET | SCHEDULED ids scored by scheduled_at (millis since epoch). |
{threadmill}:awaiting |
ZSET | AWAITING ids scored by state-entry time. |
{threadmill}:processing |
ZSET | Global PROCESSING ids scored by owner_heartbeat_at. Used by orphan recovery. |
{threadmill}:processing:{node} |
ZSET | Per-node PROCESSING ids (same score). Lets touchOwnerHeartbeat rescore one ZSET, not scan globally. |
{threadmill}:by_handler:{handler} |
SET | Members are job ids. Powers findByHandlerSignature. |
{threadmill}:by_state_time:{STATE} |
ZSET | Ids scored by current_state_at. Used for retention. |
{threadmill}:storage_format |
STRING | Completed index-format version (currently 2), or an incomplete migration marker. |
{threadmill}:storage_format_migration |
STRING with TTL | Offline migration lease. |
{threadmill}:by_state_time:{STATE}:ids |
ZSET | Zero-score ID order for bounded retry/workflow recovery pages. |
{threadmill}:counts |
HASH | State → cardinality. HINCRBY inside every state-changing script. Never SCARD / ZCARD for live counts. |
{threadmill}:queues |
SET | Active queue names (membership maintained by claim_commit). |
{threadmill}:queue_keys:{queue} |
HASH | Concurrency key → count of ENQUEUED keyed jobs of that key in the queue. An ordered ZSET mirror (:ordered) supplies bounded lexicographic registry pages. |
{threadmill}:queue_keys:{queue}:ordered |
ZSET | Zero-score lexicographic queue-key registry. |
{threadmill}:queue_unkeyed:{queue} |
ZSET | ENQUEUED unkeyed job ids, scored like the queue ZSET. The unkeyed claim lane never pages past keyed work. |
{threadmill}:queue_enqueued_at:{queue} |
ZSET | Every ENQUEUED job id in the queue, scored by current_state_at millis. oldestEnqueuedAt (the threadmill.queue.oldest.enqueued.age gauge and the dashboard queue view) reads its head with one ZRANGE 0 0 WITHSCORES, so the age gauge never scans the priority-ordered queue ZSET. Maintained inside the same atomic scripts as queue membership. |
{threadmill}:queue_pauses |
HASH | Paused queue → reason. |
{threadmill}:cron_tasks:ordered |
ZSET | Zero-score recurring-name order for bounded materializer pages. |
{threadmill}:cron_task_namespace:{namespace} |
SET | Cron task names owned by one reconciliation namespace. |
{threadmill}:cron_task_namespaces |
SET | Known recurring reconciliation namespaces. |
{threadmill}:nodes |
SET | Known NodeIds. |
{threadmill}:node:heartbeat:{node} |
STRING with TTL | Key existence is the heartbeat; TTL is the timeout. |
{threadmill}:lease:maintenance |
STRING | Maintenance-lease holder; refreshed via lease_acquire.lua. |
{threadmill}:no_key |
Reserved sentinel | Placeholder for an absent optional Lua KEYS entry; never stores data. |
{threadmill}:dedup:{queue}:{dedupKey} |
STRING | Dedup record. |
{threadmill}:dedup_expiry |
ZSET | Dedup record expiries; maintenance cleanup reads this. |
{threadmill}:concurrency_counters |
ZSET | Zero-score registry of counter keys for bounded idle cleanup. |
{threadmill}:concurrency:{key}:counters |
HASH | Per-key in-flight counts (exclusive_in_flight, shared_in_flight) and optional idle_since grace marker. |
{threadmill}:concurrency:{key}:pending |
ZSET | Pending concurrency members, scored by enqueue-time micros. |
{threadmill}:concurrency:{key}:pending:exclusive |
ZSET | EXCLUSIVE-only pending mirror for the admission barrier. |
{threadmill}:concurrency:{key}:pending:ready:{queueKeys} |
ZSET | Per-queue ENQUEUED pending mirror; {queueKeys} is the full encoded queue-key registry key. |
{threadmill}:concurrency:{key}:pending_root:{root} |
ZSET | Per workflow-root mirror of pending (same members and scores), kept only for members whose workflow root differs from their own job id. Lets the claim path find active-hold members without scanning the pending population. |
{threadmill}:concurrency:{key}:workflows |
HASH | Workflow root id → active outstanding hold count. Presence means the workflow currently owns the key. |
{threadmill}:concurrency:{key}:workflow_counts |
HASH | Workflow root id → total non-terminal job count. Maintained incrementally so claim does not scan active jobs. |
{threadmill}:concurrency:{key}:claim_lock |
STRING | Short-lived mutex around per-key claim bookkeeping. |
Development reset
Section titled “Development reset”RedisJobStore.dropThreadmillKeys() deletes every key matching
{threadmill}:* and leaves non-Threadmill Redis keys alone. It is intended for
local or ephemeral environments where deleting all Threadmill jobs, recurring
tasks, dedup records, queue pauses, leases, heartbeats, and concurrency
bookkeeping is acceptable.
Spring Boot can run the same reset before creating an auto-configured Redis store:
threadmill: store: redis: reset-on-start: true allow-destructive-reset: trueStop other Threadmill nodes before using this in a shared Redis deployment. This is not a production cleanup or migration mechanism.
Lua script inventory
Section titled “Lua script inventory”Located under src/main/resources/com/hemju/threadmill/store/redis/lua/,
loaded once at startup by LuaScripts. Each script is atomic at the Redis
server (single-threaded execution).
| Script | What it does |
|---|---|
insert.lua |
Atomically insert one job: HSET the hash, ZADD active index, ZADD by-state-time, SADD by-handler, HINCRBY counts. EXISTS-check guards against duplicates. |
insert_all.lua |
Atomic batch insert. Two-pass: first EXISTS-checks every job id; if all clear, applies every HSET / ZADD / HINCRBY in one script. Either every row lands or none do. |
enqueue_if_absent.lua |
Producer-side dedup: insert iff (queue, dedupKey) isn’t already mapped to an active job. |
save_atomic.lua |
Version-matched conditional update — the optimistic-lock save. |
claim_commit.lua |
The reliable-fetch claim. Java prepares the PROCESSING body first, then this script verifies version / state / queue membership and commits body, scalars, indexes (queue → processing + per-node), attempts, owner heartbeat, and counts together. Consults concurrency counters, pending members, workflow counts, and workflow holds before committing. A crash before this script leaves the job ENQUEUED; a crash after leaves a complete PROCESSING record for orphan recovery. |
touch_heartbeat.lua |
Explicit owner-wide heartbeat helper for external callers. Does not bump optimistic-lock version. |
touch_execution_heartbeats.lua |
Engine heartbeat: refresh at most 500 confirmed ID/version/owner-matching PROCESSING attempts; unreturned claims can expire into recovery. |
replace_job.lua |
Atomic in-place definition swap for non-running jobs. Moves the row between queue ZSETs if the queue changes. |
soft_delete.lua |
Move a job to DELETED, removing it from active indexes and per-handler set, decrementing counts. |
mutex_acquire.lua |
Acquire-or-refresh a named mutex with a millisecond-precision lease. One Lua call removes the race window that SET NX + PEXPIRE would have. |
lease_acquire.lua |
Compare-and-renew for the maintenance lease. |
lease_release.lua |
Compare-and-delete for the maintenance lease. |
dedup_delete.lua |
Compare-and-delete an expired dedup record without erasing a concurrent replacement. |
retention_candidates.lua |
Read-only cutoff-eligible time/ID paging, with bounded seeking through timestamp ties even after cursor deletion. |
cleanup_concurrency.lua |
Reclaim idle counters after a one-minute grace, preserving active and pending work. |
retention_delete.lua |
State-checked hard deletion with atomic index and count cleanup. |
queue_prune.lua |
Remove an empty queue from the registry without racing a concurrent insert. |
Reliable-fetch claim
Section titled “Reliable-fetch claim”Never a destructive BLPOP / ZPOPMIN. The flow is:
- Java gathers unkeyed heads and bounded windows from each selected key’s queue-specific ready ZSET. A lexicographic cursor pages the ordered queue-key registry; per-key windows rotate so active workflow members remain reachable behind blocked heads. Reads are pipelined, then candidates sort by priority and UUID. No global pending window is filtered after truncation, and no pass enumerates every active workflow hold.
- For each candidate, Java prepares the new body with the
PROCESSINGstate-history entry appended and the version bumped. claim_commit.luaverifies version / state / queue membership plus the concurrency admission rules (it is the single admission authority — the gathering reads are unlocked approximations). It reads only the earliest pending or earliest EXCLUSIVE member, ordered by score thenid:MODE, and commits the new body + every index update + counts in one atomic call.
A crash before step 3 leaves the job in ENQUEUED. A crash after step 3 leaves a complete PROCESSING record for the orphan-recovery path.
Short per-key preparation locks have a 30-second expiry as a crash fallback. A timed-out or interrupted lock acquisition may have committed in Redis, so cleanup compares the acquisition token before deleting it. Partial bulk acquisitions and failed workflow re-reads release every acquired lock; one cleanup error does not skip the other locks or replace the acquisition failure. Cleanup temporarily clears and then restores the caller’s interrupt flag so Lettuce can complete its bounded release. A continuing outage can still prevent cleanup; expiry remains the fallback. This does not change at-least-once delivery.
Capabilities
Section titled “Capabilities”supportsRichSearch = false. Redis cannot do deep ad-hoc metadata search;
features that need it should skip-or-degrade on Redis. supportsExactCounts
and supportsConcurrencyGroups are both true.
Lettuce / connection management
Section titled “Lettuce / connection management”Default client is Lettuce. The store either owns the client (constructed
from a RedisURI or RedisStoreConfig) or borrows one (passed in by the
host). close() shuts down the connection; the client is shut down too
only if the store owns it.
Lua return-value conventions
Section titled “Lua return-value conventions”Mixed-type Lua returns confuse Lettuce’s CommandOutput. Pick one shape
per script:
- Always-string (
OK/STALE/EXISTS/ …) →ScriptOutputType.VALUE. - Always-int →
ScriptOutputType.INTEGER. - Always list of strings →
ScriptOutputType.MULTI.
./gradlew :threadmill-store-redis:testRuns the shared store contract and backend regressions against real
redis:7.4-alpine Testcontainers. The Gradle test report is the source for
current executed, skipped, and failed test counts.
Admission index format 2
Section titled “Admission index format 2”The global pending ZSET uses id:MODE members, scored in microseconds. Its
:exclusive mirror contains only EXCLUSIVE members. Its
:ready:<queue-keys-key> mirror contains only ENQUEUED members of one queue.
The queue-key count hash has a :ordered ZSET mirror. All these indexes change
atomically with the job through the shared pending_indexes.lua helpers.
Registry pages contain at most 256 keys. Per-key candidate windows divide the claim budget across those keys; a shrinking window may require one extra read to wrap. A claim call makes at most 20 passes. Cursor caches retain 1,024 active entries; eviction resets scan progress and can increase delay for cold queues or keys. These bounds constrain each call; they are not a latency SLA under unbounded queue/key growth. One namespace still occupies one Redis Cluster slot; Cluster support provides topology and failover integration, not horizontal sharding of that namespace.
Existing v0.3 data requires the offline upgrade documented in Redis topologies.
searchJobs accepts a state with pagination. It rejects state-less requests and
queue/handler filters with IllegalArgumentException, matching the advertised
supportsRichSearch=false capability and the dashboard’s existing validation.
It never returns a filtered fragment of an unrelated global page. State-only
pages preserve the Redis index order: newest transition millisecond first, then
descending canonical job ID for ties. The order is independent of page size;
concurrent state changes can still move records between pages.