Skip to main content

Online edition for BLOGE 0.9.8-RC1 · facts verified 2026-09-15 · 中文

Chapter 6 — Resilience by Design

Promise: By the end of this chapter you will know how BLOGE's built-in resilience chain — retry → timeout → fallback — protects each node, and how compensation lets you undo already-completed work when a graph fails downstream.

Learning Goals​

  1. Explain the resilience evaluation order and why retry wraps per-attempt timeout while fallback is the final degradation.
  2. Configure retry, timeout, and fallback on a node and predict the runtime behaviour when the operator throws.
  3. Choose the right backoff strategy (fixed, exponential, jitter) for a given scenario.
  4. Declare compensate blocks to express best-effort, node-level saga cleanup — and understand why this is not a distributed-transaction rollback.

Prerequisites​

Source Examples​

FileWhat it shows
ch06/error-handling.blogeRetry + timeout + fallback on a payment node with a summarise step
ch21/retry-with-backoff.blogeLoop-based retry with custom exponential backoff and carry state
ch21/order-saga.blogeThree-node saga with per-node compensation
retry-fallback-output.blogeConformance fixture: retry + fallback on a minimal node
node-compensation.blogeConformance fixture: single-node compensation
compensation.blogeConformance fixture: two-node compensation chain
ADR-007Architecture decision record for the Retry → Timeout → Fallback order

Why This Matters​

In Chapters 1–4 you saw retry, timeout, and fallback on a few nodes. They looked like simple annotations. In reality, the order these policies execute in determines whether your graph recovers cleanly or masks real failures.

Consider a payment node that is flaky under load:

  • If fallback ran before retries were exhausted, you would silently degrade on the first hiccup — charging no one and generating phantom "manual review" records.
  • If a single timeout wrapped all retry attempts, one slow attempt could eat the entire budget, leaving no room for the remaining tries.
  • If compensation tried to act like a distributed transaction, a single compensation failure would halt cleanup and leave your system half-undone.

BLOGE eliminates these traps with two clear rules:

  1. Retry wraps per-attempt timeout; fallback is the final catch.
  2. Compensation is best-effort node-level cleanup, not an atomic rollback.

Once you internalise these rules you can declare resilience next to the node it protects and trust the engine to apply it correctly.

Mental Model​

The resilience chain​

Diagram: 06-resilience-by-design figure 1

The same chain as a rendered diagram:

Diagram: 06-resilience-by-design figure 2

Read the diagram inside-out:

  1. Operator — your business code runs first.
  2. Timeout — each individual attempt gets its own clock. If it expires, the attempt fails with OperatorTimeoutException.
  3. Retry — on failure, the engine waits the backoff duration and tries again, up to attempts times. Each new attempt starts a fresh timeout.
  4. Fallback — only if every attempt has failed (including timeouts) does the engine substitute the fallback value. This is the last resort, not an early exit.

Key insight: Overall wall-clock time can exceed a single timeout value because each retry attempt gets its own budget. For example, timeout = 1s with attempts: 3 could take up to 3 × 1 s + backoff delays.

Compensation — the undo path​

Diagram: 06-resilience-by-design figure 3

When a downstream node fails and the graph cannot continue, the engine walks back through completed nodes in reverse topological order and invokes each node's compensate operator — if one is declared. Compensation is best-effort: if ReleaseInventory itself throws, the error is captured in CompensationResult and RefundPayment still runs. This is saga-style cleanup, not distributed-transaction rollback.

First Working Example​

Here is the full ch06/error-handling.bloge:

graph errorHandlingShowcase {
  node chargePayment : ChargePaymentOperator {
    input {
      orderId         = ctx.orderId
      amount          = ctx.amount
      simulateFailure = ctx.simulateFailure
    }
    retry = { attempts: 1, backoff: 25ms, strategy: exponential }
    timeout = 1s
    fallback = { approved: false, note: "Gateway unavailable; queued for manual review" }
  }

  node summarizeOutcome : SummarizeOutcomeOperator {
    depends_on = [chargePayment]
    input {
      approved = chargePayment.output.approved
      note     = chargePayment.output.note
    }
    timeout = 1s
  }
}

Walk through the scenario where ChargePaymentOperator keeps throwing:

  1. Attempt 1 starts. The operator has up to 1 s to complete. It throws.
  2. The engine waits 25 ms (exponential backoff from the base), then starts attempt 2 with a fresh 1 s timeout. It throws again.
  3. The configured single retry is exhausted. The engine activates the fallback: { approved: false, note: "Gateway unavailable; queued for manual review" }.
  4. summarizeOutcome receives the fallback output via chargePayment.output.approved and chargePayment.output.note — the graph completes successfully with a degraded result.

Notice: summarizeOutcome has no retry or fallback. If it fails, the graph fails. Resilience is declared only where it is needed.

Java API Equivalent

The same retry + timeout + fallback configuration in the fluent Java API. The important shape is the real builder chain: lambda-based input assembly, retry(...), timeout(...), then fallback(...).

Graph graph = Graph.builder("errorHandlingShowcase")
    .node("chargePayment", new ChargePaymentOperator())
        .input((results, ctx) -> new ChargePaymentInput(
                ctx.get("orderId", String.class),
                ctx.get("amount", BigDecimal.class),
                ctx.get("simulateFailure", Boolean.class)))
        .retry(1, Duration.ofMillis(25), BackoffStrategy.EXPONENTIAL)
        .timeout(Duration.ofSeconds(1))
        .fallback(ex -> new PaymentResult(
                false,
                "Gateway unavailable; queued for manual review"))
    .node("summarizeOutcome", new SummarizeOutcomeOperator())
        .dependsOn("chargePayment")
        .input((results, ctx) -> {
            PaymentResult payment = results.get("chargePayment", PaymentResult.class);
            return new SummaryInput(payment.approved(), payment.note());
        })
        .timeout(Duration.ofSeconds(1))
    .build();

Notice the builder mirrors the DSL declaration order: retry, then timeout, then fallback. The engine applies them in the same Retry → Timeout → Fallback chain regardless of how the graph is constructed.

The Failure That Looks Safe: Charged After Timeout​

Suppose ChargeCardOperator sends a request with idempotencyKey=order-42. The payment service commits the charge at 430 ms, but its response is delayed. At 500 ms the graph times out and marks the node failed. "The node failed" and "the charge did not happen" are different facts.

Diagram: payment commits before the timeout response

Separate execution control from effect truth​

ObservationWhat it establishesWhat it does not establish
Node status is FAILED with timeoutThe engine stopped waiting within its policyThe remote system rolled back
Payment ledger contains order-42The charge effect existsThe graph received the response
A retry returns the same charge idThe provider honoured the idempotency keyEvery provider action is generally safe to retry

The RC1 foundation probe makes the smallest version of this problem observable: the fake ledger increments before the Operator response is delayed; the node then reaches FAILED, while the ledger remains at 1. This does not emulate a payment network. It proves the narrower point that engine timeout is not evidence of effect absence.

Design the recovery conversation​

For effectful Operators, define three things before adding retry:

  1. A stable idempotency key derived from the business operation, not the attempt number.
  2. A reconciliation read that can answer "did this effect happen?" after an ambiguous response.
  3. A compensation action with its own failure handling when the business truly needs an undo.

Retry is safe only inside that contract. Compensation is a new effect, not a time machine, and can fail independently. The engine controls attempts and timing; the integration owns the truth of the external ledger.

Break It Apart​

Retry and backoff strategies​

retry = { attempts: 3, backoff: 200ms, strategy: exponential }
FieldMeaning
attemptsMaximum number of retries after the initial execution (so attempts: 3 means up to 4 total executions)
backoffBase delay between retries
strategyHow the delay grows: fixed, exponential, or jitter

The three strategies:

StrategyBehaviourGood for
fixedSame delay every timeSimple cases, predictable latency
exponentialDelay doubles each retry (25 ms → 50 ms → 100 ms …)Back-pressure on overloaded services
jitterExponential with random perturbationAvoiding thundering-herd retries from parallel nodes

Fallback​

fallback = { approved: false, reason: "credit service unavailable" }

A fallback is a static value that replaces the node's output when all retry attempts and timeouts have been exhausted. It is the final degradation — the graph continues, but with substitute data.

The conformance fixture retry-fallback-output.bloge shows the minimal form:

graph g {
  node a : Op {
    retry = { attempts: 3, backoff: 200ms, strategy: exponential }
    fallback = { score: 0 }
    output {
      result: String
    }
  }
}

That fixture is intentionally tiny and focuses on fallback syntax. In a real workflow, make sure the fallback payload matches the node's declared output contract.

Compensation​

Compensation is declared inside the node that produced the side effect:

node reserveInventory : ReserveInventoryOperator {
  output {
    reservationId: String
  }

  compensate : ReleaseInventoryOperator {
    input {
      reservationId = reserveInventory.output.reservationId
    }
  }
}

(From ch21/order-saga.bloge.)

Key rules:

  • Only completed nodes are compensated. If chargePayment never finished, there is nothing to undo.
  • Reverse topological order. The engine compensates the latest completed node first and works backwards.
  • Best-effort. A compensation failure is recorded in CompensationResult but does not stop subsequent compensations.
  • Not a distributed transaction. There is no two-phase commit. Each compensate block runs independently. Design your compensation operators to be idempotent.

The conformance fixture compensation.bloge shows two nodes with compensation chaining:

graph g {
  node pay : PaymentOp {
    input { amount = ctx.amount }
    compensate : ReversePayment {
      input { txId = pay.output.transactionId }
    }
  }
  node ship : ShipOp {
    depends_on = [pay]
    compensate : CancelShipment {
      input { trackingId = ship.output.trackingId }
    }
  }
}

If ship fails, the engine compensates pay (the only completed node) by running ReversePayment with the original transaction ID.

Common Trap​

❌ Treating compensation as a distributed-transaction rollback​

// WRONG mental model — expecting atomic all-or-nothing
node reserveInventory : ReserveInventoryOperator {
  compensate : ReleaseInventoryOperator {
    input { reservationId = reserveInventory.output.reservationId }
  }
}

node chargePayment : ChargePaymentOperator {
  depends_on = [reserveInventory]
  compensate : RefundPaymentOperator {
    input { chargeId = chargePayment.output.chargeId }
  }
}

If RefundPaymentOperator throws, the reservation is still released — the engine does not stop compensating. And if ReleaseInventoryOperator also throws, you end up with two compensation failures, not a clean rollback.

Design compensation operators to be idempotent and to tolerate partial failure. Log aggressively. Use the CompensationResult list in the GraphResult to detect and alert on compensation failures.

What Goes Wrong​

All retries exhausted, no fallback​

node chargePayment : ChargePaymentOperator {
  retry = { attempts: 3, backoff: 200ms, strategy: exponential }
  timeout = 1s
  // No fallback declared
}

If the operator fails on every attempt:

GraphResult {
  status: FAILED,
  failedNode: "chargePayment",
  error: "chargePayment failed after the initial attempt
    and all configured retries"
}

The graph stops. Every downstream node is marked NOT_REACHED.

Fix: Add a fallback when graceful degradation is acceptable, or let the failure propagate when it is genuinely unrecoverable.

Guided Rewrite​

Start from the order-saga graph and add resilience:

graph orderSaga {
  node reserveInventory : ReserveInventoryOperator {
    timeout = 3s
    retry = { attempts: 2, backoff: 100ms, strategy: jitter }
    output {
      reservationId: String
    }

    compensate : ReleaseInventoryOperator {
      input {
        reservationId = reserveInventory.output.reservationId
      }
    }
  }

  node chargePayment : ChargePaymentOperator {
    depends_on = [reserveInventory]
    timeout = 5s
    retry = { attempts: 3, backoff: 200ms, strategy: exponential }
    fallback = { chargeId: null, status: "payment_deferred" }
    output {
      chargeId: String
    }

    compensate : RefundPaymentOperator {
      input {
        chargeId = chargePayment.output.chargeId
      }
    }
  }

  node shipOrder : ShipOrderOperator {
    depends_on = [chargePayment]
    timeout = 10s
    input {
      failShipping = ctx.failShipping
    }
  }
}

Questions to consider:

  1. If chargePayment fails after all 3 retries, what happens? (The fallback value { chargeId: null, status: "payment_deferred" } is used. shipOrder receives it. No compensation runs because no node has failed the graph — the fallback kept it alive.)
  2. If shipOrder then fails (no fallback), which compensation operators run? (Both RefundPaymentOperator and ReleaseInventoryOperator, in that order — reverse topological. chargePayment completed with the fallback value, and reserveInventory completed normally.)
  3. What is the worst-case wall-clock time for chargePayment alone? (Initial attempt: 5 s + 3 retries × (backoff + 5 s). With exponential backoff from 200 ms: ≈ 5 + 0.2 + 5 + 0.4 + 5 + 0.8 + 5 = ~21.4 s.)

Brain Check​

  1. What is the resilience evaluation order in BLOGE? (Retry → Timeout (per attempt) → Fallback.)
  2. Why does each retry attempt get its own timeout rather than sharing one global timeout? (So that one slow attempt cannot consume the entire budget, leaving no time for subsequent retries. See ADR-007.)
  3. When does a fallback activate? (Only after all retry attempts — including their individual timeouts — have been exhausted. It is the final degradation.)
  4. Does a compensation failure stop the engine from compensating earlier nodes? (No. Compensation is best-effort. Each failure is captured in a CompensationResult and the engine continues to the next node.)
  5. What order does the engine compensate nodes in? (Reverse topological order, only for nodes whose primary execution completed successfully.)
  6. Design question: Your graph calls three external payment gateways in a fan-out. Each has retry + fallback. If all three fallback, the downstream aggregation node receives three degraded results. How should you design the aggregation node to handle a mix of real and fallback data? (The aggregation operator should inspect a flag or sentinel value in each input — for example, a "status": "degraded" field from the fallback — and decide whether to proceed with partial data or escalate. The graph-level design decision is whether "all degraded" should still complete or should fail the graph.)

Lab​

  1. Open ch06/error-handling.bloge.

    • Change the retry strategy from exponential to jitter and increase attempts to 3.
    • Predict: if every attempt takes exactly 800 ms before failing, will the 1 s timeout ever fire? (No — 800 ms < 1 s, so each attempt fails from the operator exception, not from timeout.)

  2. Open ch21/order-saga.bloge.

    • Add timeout = 2s and retry = { attempts: 1, backoff: 50ms, strategy: fixed } to chargePayment.
    • Add a fallback to reserveInventory that returns { reservationId: "NONE" }.
    • Trace what happens when reserveInventory's operator always throws: does ReleaseInventoryOperator run? (No — the fallback kept reserveInventory alive, so the graph continues. Compensation only triggers if the graph ultimately fails.)

  3. Write a new graph with three independent service calls, each with different backoff strategies (fixed, exponential, jitter). Add a fan-in node that requires all three. Give two of the three a fallback. Predict: under what conditions does the graph fail vs. degrade gracefully?

Experiment acceptance card​

  • Expected and observed: The graph can fall back after timeout while the external charge may already exist.
  • Failure and recovery: Delay the effect beyond timeout; recover with an idempotency key and reconciliation.
  • Proof boundary: Proves node completion and external effect are separate facts, not exactly-once.
  • Exercise contract: Charge probe; change only response delay; deliver graph status and ledger count; stop when both facts are separate.

Recap​

  • BLOGE applies resilience in a strict order: Retry → Timeout → Fallback. Each retry attempt gets its own timeout budget; fallback is the final degradation after all attempts are exhausted.
  • Three backoff strategies control delay between retries: fixed, exponential, and jitter.
  • Fallback is a static substitute value — the graph continues with degraded data instead of failing.
  • Compensation (compensate blocks) provides best-effort, node-level saga cleanup. The engine runs compensation in reverse topological order, only for completed nodes, and captures failures without halting.
  • Compensation is not a distributed-transaction rollback. Design compensation operators to be idempotent.
  • Resilience is declared on the node, not buried in business code — the engine owns the retry loop, the timeout clock, and the compensation walk.

Next Step​

In Chapter 7 — Designing Good Operators you will learn how to write operators that are testable, composable, and play well with the resilience and scheduling features you have seen so far.

Coding Agent: Open the versioned task guide.