Skip to main content

Online edition for BLOGE 0.9.8-RC1 · facts verified 2026-09-15 · 中文

Chapter 22 — Distributed Runtime and Capacity Validation

Promise: You will place the same order graph behind two workers sharing durable stores, explain why leases need fencing and routes need stable bindings, then produce a capacity claim that applies only to an explicit workload and environment.

Learning goals​

By the end of this chapter, you can:

  1. Explain how lease owner, timeout, and epoch make a takeover safe.
  2. Separate stable execution routing from a shard resolver's new choice.
  3. Locate the resource that saturates before adding workers.
  4. Design a minimum capacity experiment around arrival rate, service time, concurrency, and backlog.
  5. State the distributed contract a test proves and its external boundary.

This chapter adds one variable​

The previous chapter fixed the world inside one JVM. The order graph is unchanged, but Worker A and Worker B now share durable stores. One variable is new: process memory no longer guarantees execution ownership.

Two hazards follow: A writes after being considered dead; the same business key drifts to another shard because routing policy changed.

Opening problem: can B continue as soon as A times out​

Predict whether this timeline is safe:

10:00:00  Worker A claims order-42, epoch=1
10:00:20  A writes a checkpoint
10:01:01  lease expires; Worker B claims, epoch=2
10:01:03  A's network returns; A writes again

If a store checks only an owner name, A can overwrite B's newer state. Safe takeover needs both timeout and a monotonically increasing fencing epoch.

Figure 22.1: lease expiry advances the epoch and fences the old owner

Run two RC1 distributed-contract tests​

cd submodule/bloge
mvn -pl bloge-durable \
  -Dtest=ClusterFailoverSimulationTest,RuntimeRoutingShardingTest test

Observed on 2026-09-14 at commit cc38fbe5:

RuntimeRoutingShardingTest    Tests run: 2, Failures: 0
ClusterFailoverSimulationTest Tests run: 1, Failures: 0
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0

These tests use a logical clock and in-memory stores. They establish deterministic contracts, not database throughput.

Mechanism one: takeover is a three-condition chain​

ClusterFailoverSimulationTest performs four actions:

  1. A claims an execution at epoch 1 and writes a checkpoint in that lease context.
  2. Logical time moves beyond the one-minute timeout.
  3. B's recovery loop reclaims at epoch 2, writes a recovery marker, and completes the execution.
  4. A tries to write with epoch 1; the store throws StaleFencingEpochException.

Safe takeover requires all three conditions:

lease expired
  + new owner atomically claims and advances epoch
  + every protected write validates current epoch

Without the third, a zombie worker can corrupt state. Without a recovery scan, fencing alone leaves the execution stranded.

Single-factor break: accept a stale epoch​

Keep timeout, checkpoints, and recovery unchanged; only allow A's final epoch-1 write. Final status may still say COMPLETED, while the old owner overwrites the recovery marker. Equal final status does not prove a correct takeover.

Recovery means carrying lease context on every protected mutation and failing closed in persistence. Logs cannot repair the conflict afterward.

Mechanism two: routing is a stable business-key binding​

RuntimeRoutingShardingTest first persists:

tenant-a + order-999 → shard-a

Even when a second engine's resolver returns shard-b, the existing binding wins and the second execution stays on shard-a.

Figure 22.2: new requests choose a shard; existing business keys reuse their binding

The resolver answers “where should an unbound key go?” The routing store records “where does this business object already live?” Rehashing on every request can lose recovery and event correlation after the shard count changes.

Migration must be an explicit control-plane operation: freeze the key, move state, update the binding, verify reads and writes, then resume traffic. It is not ordinary resolver recomputation.

Capacity: find the bottleneck before scaling it​

Think of throughput as a narrow pipe:

arrival rate
  → runnable worker concurrency
    → external API quota
      → connection pool
        → durable-store writes and lock contention

Adding workers widens only the second segment. If checkpoint latency is saturated, more workers add queueing and lock pressure.

Little's Law gives a first estimate: in a stable system, in-flight ≈ arrival rate × average service time. It helps design measurement; it is not a BLOGE capacity guarantee.

MeasureMeaning
arrival ratenew executions or work items per second
service timedistribution from runnable to complete
in-flightwork currently holding workers/resources
backlog ageage of the oldest waiting item
store latencyp50/p95/p99 of checkpoint, claim, and poll

An attributable scaling experiment​

Fix graph, input distribution, database, connection pool, and external stubs. Change only worker count from 1 to 2, 4, and 8. Record throughput, p95, backlog age, and store latency.

  • Throughput grows while store latency stays flat: workers may be the current bottleneck.
  • Throughput stays flat while store latency rises: store or lock contention is more likely.
  • Backlog falls while downstream 429s rise: pressure was merely moved.

Change one axis per experiment or the worker-count effect is not attributable.

A remote worker is a dispatch boundary, not free capacity​

WorkerDispatcher sends selected work items to a remote pool; the Kafka module provides one transport. Dispatch adds serialization, delivery, duplicates, heartbeat, result correlation, and cancellation semantics. A GPU worker may solve hardware affinity but does not create exactly-once effects.

The main chapter keeps only that boundary. Topic, heartbeat, batch-size, schema-migration status, and benchmark parameters live in the runtime reference so architectural judgment is not buried in a configuration catalog.

Real-world transfer: takeover in a warehouse network​

Warehouse A scans a parcel and loses connectivity. Warehouse B must not ship merely because A has been silent for a minute. Headquarters issues a newer work token and rejects A's old token. An order already bound to the east warehouse must not drift just because a south warehouse was added today.

Those are fencing epoch and routing binding. More warehouses help only when picking stations are the bottleneck. If the central inventory lock is saturated, more sites increase contention.

Your turn: deliver a capacity conclusion, not a chart​

Choose one durable graph and define:

  1. One fixed input distribution and target arrival rate.
  2. Worker counts 1, 2, and 4 with all other conditions fixed.
  3. Throughput, p95, backlog age, and store p95.
  4. One worker interruption and one stale-epoch write attempt.
  5. A stop rule, such as backlog age rising for three consecutive windows.

Deliver raw results, an environment card, a single-axis comparison, and one sentence: “Under these conditions, this resource saturated first.” Do not write “production traffic is supported” unless the production traffic model and external dependencies are inside the evidence boundary.

What this chapter proved—and did not prove​

The focused tests prove that, in in-memory simulations, a new owner reclaims an expired execution at a higher epoch, stale writes are rejected, and an existing business-key binding beats a new resolver choice. They do not prove real-database isolation, Kafka delivery semantics, cross-region recovery, or production capacity.

The next stage begins with Chapter 23 — Business Correctness Is Not Test Pass: it moves from bounded runtime evidence to asking whether the declared business promise is satisfied.

Experiment acceptance card​

  • Expected and observed: Takeover requires matching lease, epoch, and route while the business key stays bound.
  • Failure and recovery: Use a stale epoch; recover the new lease/epoch and reject the old owner.
  • Proof boundary: Proves in-memory takeover and routing, not database isolation or capacity.
  • Exercise contract: One failover; change only epoch; deliver takeover and route receipts; stop when stale writes fail and one owner remains.

Exact fact entry points​

Coding Agent: Open the versioned task guide.