threadmill-simulation
End-to-end correctness verification under realistic mixed workload and
worker-process churn. Not a performance benchmark (that’s
threadmill-soak), not a teaching example (that’s threadmill-example) — a
third thing: simulations that exercise load-bearing primitives and produce
JSON-lines traces for verification or review.
Correctness Simulation
Section titled “Correctness Simulation”This is the short fixed invariant verifier. It runs in seconds, uses Testcontainers for Postgres, Oracle, and Redis, and fails the Gradle task when a backend does not drain or the trace verifier finds an invariant violation.
./gradlew :threadmill-simulation:simulateRuns all four backends sequentially (in-memory, Postgres 18 via
Testcontainers, Oracle Database 23ai Free via Testcontainers
gvenzl/oracle-free:23-slim-faststart, Redis via Testcontainers). One command,
full validation. Or pick one:
./gradlew :threadmill-simulation:simulateMemory # fast, no Docker./gradlew :threadmill-simulation:simulatePostgres # Testcontainers./gradlew :threadmill-simulation:simulateOracle # Testcontainers./gradlew :threadmill-simulation:simulateRedis # TestcontainersWhat the Scenario Does
Section titled “What the Scenario Does”50 projects, each with its own queue (project:1 through project:50). Two
job shapes per project:
- Import —
EXCLUSIVElock onproject:N, slower (30 ms simulated). - Export —
SHAREDlock onproject:N, faster (8 ms simulated).
Mix: ~10% imports, ~90% exports. 400 jobs total — small enough to finish in
seconds, big enough to exercise contention. Half the jobs are enqueued
one-by-one via Scheduler.enqueue(...); the other half via
JobStore.insertAll(...) so the bulk-insert path runs under contention
with the engine already draining work.
Failure injection per attempt:
- 5% throw a
RuntimeException(retry path). - 0.5% hang past
jobTimeout(timeout path).
Mid-run, a randomly-selected project is paused via
Scheduler.pauseQueue(...) and resumed after a moment.
What the Verifier Asserts
Section titled “What the Verifier Asserts”Every JSON-lines event in build/simulation/*.jsonl is read back into
TraceVerifier. Violations are reported with the exact event chain that
proves them. Invariants:
- At-least-once delivery. Every enqueued job has a terminal event:
succeeded,quarantined, orfailed/timed_outwithfinal=true. - Concurrency exclusion. For any concurrency key, an
EXCLUSIVElock_acquirednever overlaps with another lock for the same key; aSHAREDlock_acquirednever overlaps with anEXCLUSIVEfor the same key. - Lock pairing. Every
lock_acquiredhas a matchinglock_releasedfor the same job id and key. - Pause obeyed. Between
queue_pausedand the matchingqueue_resumedfor one queue, noclaimedevent names that queue.
Trace Format
Section titled “Trace Format”JSON-lines (*.jsonl). One event per line. Every line is a complete JSON
object with at least timestamp (ISO-8601 UTC) and event. Common fields:
{ "timestamp": "…", "event": "enqueued", "jobId": "…", "queue": "…", "type": "import|export", "lockKey": "…", "lockMode": "EXCLUSIVE|SHARED" }{ "timestamp": "…", "event": "claimed", "jobId": "…", "queue": "…", "attempt": 1 }{ "timestamp": "…", "event": "lock_acquired", "jobId": "…", "lockKey": "…", "lockMode": "…" }{ "timestamp": "…", "event": "succeeded", "jobId": "…", "queue": "…", "attempts": 2, "final": true }{ "timestamp": "…", "event": "failed", "jobId": "…", "queue": "…", "attempts": 1, "causeKind": "EXCEPTION", "causeMessage": "…", "final": false }{ "timestamp": "…", "event": "timed_out", "jobId": "…", "queue": "…", "attempts": 2, "causeKind": "TIMEOUT", "final": true }{ "timestamp": "…", "event": "lock_released", "jobId": "…", "lockKey": "…", "lockMode": "…" }{ "timestamp": "…", "event": "queue_paused", "queue": "…" }{ "timestamp": "…", "event": "queue_resumed", "queue": "…" }{ "timestamp": "…", "event": "node_started", "nodeId": "…", "workerCount": 8, "backend": "memory|postgres|oracle|redis" }{ "timestamp": "…", "event": "node_stopped", "nodeId": "…" }Exit Code
Section titled “Exit Code”The Gradle task fails (non-zero exit) when any backend either:
- didn’t fully drain inside the run budget, or
- produced a trace with at least one verifier violation.
Adding A New Scenario
Section titled “Adding A New Scenario”Add a new payload type and handler in SimulationPayloads, register it in
SimulationRunner.enqueueWorkload. If your invariant isn’t on the
verifier’s existing list, add a new check method in TraceVerifier and
document it above.
Worker-Churn Simulation
Section titled “Worker-Churn Simulation”This is the long-running multi-process simulation. A supervisor starts worker
JVMs against one shared external datastore, enqueues jobs continuously, kills
and recreates workers, and writes a JSON-lines trace under
build/simulation/worker-churn-<timestamp>-<backend>.jsonl by default.
It covers PostgreSQL and Redis; the Oracle store is exercised by the short
correctness simulation above and the soak suites, not by worker churn.
Start local shared datastores from the repository root:
docker compose up -d postgres redisDefaults:
- Postgres:
jdbc:postgresql://localhost:55432/threadmill - Redis:
redis://localhost:56379/0
Override with:
THREADMILL_JDBC_URLTHREADMILL_DB_USERTHREADMILL_DB_PASSWORDTHREADMILL_REDIS_URI
Postgres, 10 minutes:
./gradlew :threadmill-simulation:simulateWorkerChurnPostgres --args="--backend postgres --duration 10m --workers 4 --jobs-per-second 20 --kill-every 30s --drain-timeout 3m"Redis, 10 minutes:
./gradlew :threadmill-simulation:simulateWorkerChurnRedis --args="--backend redis --duration 10m --workers 4 --jobs-per-second 20 --kill-every 30s --drain-timeout 3m"For an overnight run, use --duration 8h. To disable forced process kills,
use --kill-every 0. Pass --trace path/to/trace.jsonl to choose a custom
trace location, or --queue name when multiple supervisors should share one
queue deliberately.
The trace contains supervisor start/stop, worker process starts, exits, kills,
and restarts, every job enqueue, every attempt start and finish, interrupted
attempts, and periodic store snapshots. It is a production-behaviour trace
generator, not a replacement for the contract tests. The task exits non-zero
when the queue does not drain within --drain-timeout.
Process-Separated Nudge Simulation
Section titled “Process-Separated Nudge Simulation”This fixed simulation starts real Postgres and Redis datastores with Testcontainers, then runs the maintenance leader, standby, and producers in separate JVMs. It covers PostgreSQL and Redis only; Oracle is not yet part of this simulation. Run both backends:
./gradlew :threadmill-simulation:simulateNudgeThe combined real-backend task is also part of the root productionCheck
release-candidate gate so the crash paths cannot silently rot.
Or select one:
./gradlew :threadmill-simulation:simulateNudgePostgres./gradlew :threadmill-simulation:simulateNudgeRedisThe leader first registers an exclusive one-minute recurring outbox pump, with its own maintenance poll set to five minutes, so process startup and lease handoff cannot race the backstop or the leader’s next tick. After failover is proved, the supervisor edits the backstop to eight seconds for the producer crash phase. It verifies two crash windows:
- A producer commits durable work and an accepted nudge. The supervisor
hard-kills the maintenance leader before its next materializer tick; the
standby must acquire the expired maintenance lease and run the pump with
origin=nudge. - A second producer commits durable work, stops immediately before the nudge,
and is hard-killed. No nudge may appear for that row; the regular recurring
fire must run with
origin=scheduleand drain it.
The verifier reads a cross-process JSON-lines trace and proves the event
ordering, distinct process ids, standby ownership, trigger origins, and final
drain. Artifacts live under
build/simulation/nudge-cross-node-<timestamp>-<backend>/, including
trace.jsonl and one output log per child JVM.
Why a separate module
Section titled “Why a separate module”threadmill-soak is about sustained load and operational performance
artifacts. threadmill-example is about teaching users how to wire
Threadmill. This module is doing a third thing: correctness verification and
worker-process churn under realistic conditions.