P(all steps succeed) = p^n
where p = per-step success rate, n = number of steps

p = 0.99
  n = 10  →  0.99^10  =  0.904  (~90%)
  n = 50  →  0.99^50  =  0.605  (~60%)
  n = 100 →  0.99^100 =  0.366  (~37%)
  n = 200 →  0.99^200 =  0.134  (~13%)

async function retryTransient<T>(
  fn: () => Promise<T>,
  maxAttempts = 3,
  baseDelayMs = 500,
  maxDelayMs = 8000
): Promise<T> {
  let lastError: unknown;
  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (e) {
      lastError = e;
      if (!isTransient(e) || attempt === maxAttempts - 1) throw e;
      // Full jitter: uniform random between 0 and the exponential cap.
      const cap = Math.min(maxDelayMs, baseDelayMs * 2 ** attempt);
      const delay = Math.random() * cap;
      await sleep(delay);
    }
  }
  throw lastError;
}

// Stripe-style idempotency key on a non-idempotent tool.
async function chargeCard(input: ChargeInput): Promise<ToolResult> {
  const key = input.idempotencyKey ?? hashStable(input);
  const cached = await idempotencyStore.get(key);
  if (cached) return cached;          // already applied — return prior result
  const result = await stripe.charges.create(input, { idempotencyKey: key });
  await idempotencyStore.set(key, result);
  return { ok: true, content: result };
}

interface Turn {
  tool_name: string;
  input_hash: string;
  output_hash: string;
  ok: boolean;
}

function detectStuckLoop(recentTurns: Turn[], threshold = 3): boolean {
  if (recentTurns.length < threshold) return false;
  const last = recentTurns.slice(-threshold);
  const sameInput = last.every(t => t.input_hash === last[0].input_hash);
  const sameOutput = last.every(t => t.output_hash === last[0].output_hash);
  const sameTool = last.every(t => t.tool_name === last[0].tool_name);
  const allFailed = last.every(t => !t.ok);
  return sameTool && sameInput && sameOutput && allFailed;
}

class CircuitBreaker {
  private disabled = new Map<string, { until: number; reason: string }>();

  trip(toolName: string, reason: string, sessionOnly = true): void {
    const until = sessionOnly ? Number.MAX_SAFE_INTEGER : Date.now() + 60_000;
    this.disabled.set(toolName, { until, reason });
  }

  isCallable(toolName: string): boolean {
    const e = this.disabled.get(toolName);
    if (!e) return true;
    if (Date.now() >= e.until) { this.disabled.delete(toolName); return true; }
    return false;
  }

  gate(toolName: string): ToolResult | null {
    const e = this.disabled.get(toolName);
    if (!e || Date.now() >= e.until) return null;
    return {
      ok: false,
      retryable: false,                          // do NOT retry — circuit is open
      error: `tool '${toolName}' disabled: ${e.reason}. Use a different approach or report you cannot proceed.`,
    };
  }
}

def handle_budget_exhaustion(session, exc: BudgetExhausted):
    checkpoint = session.checkpoint()              # atomic write — Module 8.2
    handoff = write_handoff(                       # human-legible resume — Module 4.2
        task=session.task,
        completed=session.completed_steps(),
        pending=session.pending_steps(),
        next_step=session.next_step(),
        blockers=session.blockers(),
        checkpoint_path=checkpoint.path,
    )
    return FailureReport(
        kind="budget_exhausted",
        message=f"Budget exhausted at step {session.current_step}. "
                f"State saved to {checkpoint.path}. Handoff at {handoff.path}. "
                f"Resume with: agent resume {handoff.path}",
        completed=session.completed_steps(),
        partial_artifacts=session.artifacts(),
    )

# Module 7 — Error Handling and Recovery

**Course**: Master Course · **Module**: 7 · **Duration**: 45 min · **Prerequisites**: Modules 1–6

> *Error compounding math: 10-step process at 99% per-step = 90% success. At 50 steps = 60%.*

---

## Learning Objectives

1. Apply the four-category error taxonomy (transient, LLM-recoverable, user-fixable, fatal).
2. Implement stuck-loop detection and the circuit-breaker pattern.
3. Design graceful degradation for budget exhaustion and partial completion.

---

# 7.1 — The Error Taxonomy

*The four categories. Get the taxonomy right and the harness handles errors correctly; get it wrong and you get infinite retries or silent failures.*

## The four categories

Every tool failure, provider error, and side-effect fault an agent encounters falls into exactly one of four categories. The category determines the response. Get the category right and the harness handles the error correctly; get it wrong and you ship either an infinite-retry loop or a silent hallucination of success.

| Error Type | Definition | Correct Response |
| --- | --- | --- |
| **Transient** | Network timeout, 503, rate limit | Retry with exponential backoff + jitter; cap at 2 attempts (Stripe) |
| **LLM-recoverable** | Wrong data type, malformed call, bad argument | Return error as tool result; let model self-correct next turn |
| **User-fixable** | Missing credential, ambiguous task, needs clarification | Interrupt loop; surface to human (Module 6) |
| **Fatal** | Unrecoverable state, sandbox crash, budget exhausted | Bubble up immediately; do not retry |

The classification is the harness's responsibility, not the model's. The model sees a string error message; the harness sees a typed exception with `retryable`, `category`, and `cause` fields. **The harness decides whether to retry, surface, or halt — the model only decides what to do next once the harness has acted.** This is why Module 2.2's "never throw, always return a structured error" rule is load-bearing: without structured errors, the harness cannot classify.

## The error-compounding math

A 10-step process at 99% per-step reliability = ~90% end-to-end success. At 50 steps = ~60%. At 100 steps = ~37%. This is why error handling is not a feature — it is the difference between a demo that works and a production system that ships. Every step's error rate compounds multiplicatively.

The math is just independence multiplied:

```
P(all steps succeed) = p^n
where p = per-step success rate, n = number of steps

p = 0.99
  n = 10  →  0.99^10  =  0.904  (~90%)
  n = 50  →  0.99^50  =  0.605  (~60%)
  n = 100 →  0.99^100 =  0.366  (~37%)
  n = 200 →  0.99^200 =  0.134  (~13%)
```

**The leverage point is per-step reliability, not step count.** Doubling per-step reliability from 99% to 99.99% over 100 steps lifts end-to-end success from 37% to 99%. A 200-step process that recovers 90% of transient failures (turning a 1% error into a 0.01% error) goes from 13% end-to-end to 98%. This is the quantitative justification for Module 7's existence: error handling is the single highest-leverage reliability intervention in a harness.

The independence assumption is conservative in practice. Real failures cluster (a flaky tool fails repeatedly), which makes the true end-to-end success *worse* than the multiplicative model predicts. The model is a lower bound on how bad things get — and it is already bad enough to justify every pattern in this module.

## Anti-patterns

- **Retrying everything** → infinite loops, resource exhaustion. A fatal error retried forever burns the entire budget. This is Module 1.2's "infinite loop problem," failure mode 2.
- **Swallowing everything** → silent failures. The model hallucinates success because it got no error feedback. The agent reports "task complete" while having done nothing.
- **Surfacing everything to user** → alert fatigue (Module 6.2). The user becomes the error handler and learns to dismiss every alert.
- **No taxonomy** → all errors treated identically. Usually means "retry everything" by default, which is the first anti-pattern.

The taxonomy is the cure. Classify the error; apply the category-specific response.

---

# 7.2 — Retry Strategies for Transient Errors

*The one category you retry. Done right, it converts a 1% error rate into a 0.01% error rate; done wrong, it produces thundering herds and rate-limit cascades.*

## Why retries work (and when they don't)

Transient errors are, by definition, recoverable in time — a rate-limit window expires, a load balancer fails over, a downstream service restarts. A retry after a short delay succeeds where the original call failed. The probability that a *truly transient* error persists across two well-spaced attempts is the product of two independent low-probability events: if the base rate is 1%, two attempts yield 0.01% effective failure.

This is the leverage from 7.1's compounding math. **A 99% reliable step becomes 99.99% reliable with a single well-placed retry.** Over 100 steps, that is the difference between 37% and 99% end-to-end success. The retry is cheap (one extra call, bounded delay); the reliability gain is multiplicative.

The catch: retries only help for *transient* errors. Retrying an LLM-recoverable error (wrong argument type) is a stuck loop — the same input fails identically every time. Retrying a fatal error (sandbox crash) burns resources pointlessly. **Retrying the wrong category is the failure mode the taxonomy exists to prevent.**

## Exponential backoff with jitter

The production retry pattern. Delay grows exponentially between attempts, and each delay is perturbed by random jitter. The exponential growth bounds total wall-clock time; the jitter prevents the **thundering herd** — many clients retrying in lockstep after a shared outage and re-overloading the service.

```typescript
async function retryTransient<T>(
  fn: () => Promise<T>,
  maxAttempts = 3,
  baseDelayMs = 500,
  maxDelayMs = 8000
): Promise<T> {
  let lastError: unknown;
  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (e) {
      lastError = e;
      if (!isTransient(e) || attempt === maxAttempts - 1) throw e;
      // Full jitter: uniform random between 0 and the exponential cap.
      const cap = Math.min(maxDelayMs, baseDelayMs * 2 ** attempt);
      const delay = Math.random() * cap;
      await sleep(delay);
    }
  }
  throw lastError;
}
```

**Full jitter** (uniform `[0, cap]`) outperforms "equal jitter" or "decorrelated jitter" under contention, per the AWS Architecture Blog's analysis of exponentially distributed backoff. The intuition: under load, you want retry arrivals *spread uniformly*, not clustered near a fixed point. Full jitter spreads them maximally.

## The retry cap

**Cap retries at 2–3 attempts.** Stripe's published idempotency guidance caps at 2; AWS SDK defaults to 3. Beyond that, the error is probably not transient (it has persisted too long), and continued retries risk a cascading retry storm. The cap is also the bound on extra latency: 3 attempts with full jitter maxes out around 1.5s of added delay, which is acceptable in an agentic loop where each turn already costs seconds.

The cap interacts with Module 1.2's error-threshold stop condition. A single transient error retried 3 times is one *logical* failure, not three — the harness should count logical failures against the threshold, not raw attempts, so that a transient blip does not trip the cascade-failure stop.

## Idempotency: the precondition for safe retries

A retried tool must be **idempotent** — calling it twice produces the same result as calling once, with no unintended side effects. This is Module 2.2's idempotency rule, restated at the retry layer.

- `read_file` — idempotent (reading twice is harmless).
- `search_codebase` — idempotent.
- `write_file` — idempotent if the content matches; problematic if the model retries with slightly different content.
- `bash` — almost never idempotent (`rm` twice is fine; `mkdir` twice errors; `git commit` twice creates two commits).
- `send_email` / `charge_card` — never idempotent. Two emails get sent; two charges hit the card.

For non-idempotent tools, retry is unsafe unless the tool itself deduplicates: a client-supplied **idempotency key** (Stripe's pattern), a content-hash check, or an "already-applied" guard. If the tool cannot be made idempotent, the retry layer must skip it — which means the transient-error recovery does not apply, and the error surfaces to the model for a different kind of recovery (or to the user).

```typescript
// Stripe-style idempotency key on a non-idempotent tool.
async function chargeCard(input: ChargeInput): Promise<ToolResult> {
  const key = input.idempotencyKey ?? hashStable(input);
  const cached = await idempotencyStore.get(key);
  if (cached) return cached;          // already applied — return prior result
  const result = await stripe.charges.create(input, { idempotencyKey: key });
  await idempotencyStore.set(key, result);
  return { ok: true, content: result };
}
```

---

# 7.3 — Stuck Loop Detection and the Circuit Breaker

*The model calling the same failing tool 10 times in a row. Detection and the circuit breaker.*

## The pattern

The model calls a tool. It fails. The model calls it again, same input. Fails again. Repeat. This is the stuck loop — and without detection, it consumes the entire iteration budget (Module 1.2) on a single unrecoverable error.

Stuck loops arise from two causes. First, **a non-transient error misclassified as retryable** — the model treats a wrong-argument error as a transient blip and hammers it. Second, **the model has no alternative** — it has decided this tool is the only path forward, and without a circuit breaker it will keep trying until the budget runs out. Both are cured by detection plus an enforced stop.

## Detection: same tool + same input + same error

The detection signal comes from Module 1.4's observability payload: `input_hash` + `output_hash`. If both match across N consecutive turns (typically 3), the loop is stuck — same input, same (error) output, no progress.

```typescript
interface Turn {
  tool_name: string;
  input_hash: string;
  output_hash: string;
  ok: boolean;
}

function detectStuckLoop(recentTurns: Turn[], threshold = 3): boolean {
  if (recentTurns.length < threshold) return false;
  const last = recentTurns.slice(-threshold);
  const sameInput = last.every(t => t.input_hash === last[0].input_hash);
  const sameOutput = last.every(t => t.output_hash === last[0].output_hash);
  const sameTool = last.every(t => t.tool_name === last[0].tool_name);
  const allFailed = last.every(t => !t.ok);
  return sameTool && sameInput && sameOutput && allFailed;
}
```

The `allFailed` check matters: a tool that succeeds identically three times in a row is not stuck, it is deterministic. The stuck signal is *failed + unchanged* — the model is hitting a wall and not adapting. Two-of-three (same input but different errors) is also a stuck signal of a different kind: the model is flailing. Both warrant tripping the breaker.

A subtler detector tracks **tool-call repetition across the whole session, not just the last N turns**: if `tool X` has been called more than K times total with an error rate above some threshold, the tool is unhealthy and should be quarantined even if no three consecutive calls were identical. This is the agent-level analogue of Module 9's flaky-test quarantine.

## The circuit breaker

When stuck-loop detection fires, the circuit breaker trips: **disable the failing tool for the rest of the session.** The model cannot call it again; it must find another approach or report that it cannot proceed.

This is the tool-level realization of the error-threshold stop condition from Module 1.2. The circuit breaker prevents one misbehaving tool from trapping the agent. The model gets a clear error — "tool X disabled for this session after N consecutive failures" — and can route around it.

```typescript
class CircuitBreaker {
  private disabled = new Map<string, { until: number; reason: string }>();

  trip(toolName: string, reason: string, sessionOnly = true): void {
    const until = sessionOnly ? Number.MAX_SAFE_INTEGER : Date.now() + 60_000;
    this.disabled.set(toolName, { until, reason });
  }

  isCallable(toolName: string): boolean {
    const e = this.disabled.get(toolName);
    if (!e) return true;
    if (Date.now() >= e.until) { this.disabled.delete(toolName); return true; }
    return false;
  }

  gate(toolName: string): ToolResult | null {
    const e = this.disabled.get(toolName);
    if (!e || Date.now() >= e.until) return null;
    return {
      ok: false,
      retryable: false,                          // do NOT retry — circuit is open
      error: `tool '${toolName}' disabled: ${e.reason}. Use a different approach or report you cannot proceed.`,
    };
  }
}
```

The error message is prompt engineering: it tells the model the tool is unavailable, that retrying will not help (`retryable: false`), and that it must change strategy. A vague "tool unavailable" leaves the model hammering the registry; a directive "use a different approach" routes it.

## Half-open recovery

A harder variant: the tool is not permanently broken, it is temporarily unhealthy (a downstream service restart). A **half-open circuit breaker** re-allows a single probe call after a cooldown period; if the probe succeeds, the breaker closes; if it fails, the breaker re-opens for another cooldown. This is the classic Hystrix pattern, adapted to agent tools. For session-scoped agents that run in minutes, full-session disable is usually simpler and sufficient; half-open matters for long-running agents.

---

# 7.4 — Graceful Degradation

*What happens when the agent can't finish? Partial completion, budget exhaustion, human-readable failure.*

## The graceful degradation taxonomy

When the agent cannot complete the task, "how does it fail" is itself a design decision with a spectrum. The four modes, from worst to best:

| Mode | Behavior | Problem |
| --- | --- | --- |
| **Silent abort** | Agent crashes or returns empty; no state saved | All work lost; user has no idea what happened |
| **False success** | Agent reports done but did not finish | Model hallucinates success; user trusts a broken result |
| **Loud abort** | Agent halts, reports failure, discards in-progress state | Completed steps are lost; user restarts from zero |
| **Graceful degradation** | Agent saves state, completes what it can, reports specifics | Value preserved; failure is actionable |

The taxonomy mirrors Module 1.2's stop conditions and Module 8's checkpointing. **Graceful degradation is the only acceptable production default.** The other three are defects: silent abort loses work, false success corrupts trust, and loud abort wastes the partial progress the agent already paid for.

## Partial task completion

The agent does as much as possible, then reports clearly what was skipped. "Completed steps 1–12 of 15. Steps 13–15 skipped: the test suite failed at step 12 and I could not resolve it. Manual intervention needed on test_auth.ts."

This is far better than two alternatives: silently failing (the agent claims success but didn't finish) or fully aborting (all 12 completed steps are lost). Partial completion preserves the value created.

The report must be **specific and falsifiable**: file paths, line numbers, the exact error, what was tried. Vague failures ("task failed") are useless; specific failures ("could not resolve type error in auth.ts:42 — the Token type is imported from two conflicting modules") are actionable. The falsifiability test: could a human pick up exactly where the agent stopped and continue? If yes, the report succeeded; if no, the agent regressed to silent abort.

## Budget exhaustion recovery

When the token or compute budget (Module 1.2, Module 5.4) is exhausted mid-task, the harness should:

1. **Save state** (checkpoint — Module 8). The current plan, completed steps, and pending work are serialized so a continuation can resume losslessly.
2. **Exit cleanly** with a clear message: "Budget exhausted at step 15. State saved. Resume with a new session."
3. **Write the handoff file** (Module 4.2) so a continuation session can resume — task state, decisions, next step, blockers.

```python
def handle_budget_exhaustion(session, exc: BudgetExhausted):
    checkpoint = session.checkpoint()              # atomic write — Module 8.2
    handoff = write_handoff(                       # human-legible resume — Module 4.2
        task=session.task,
        completed=session.completed_steps(),
        pending=session.pending_steps(),
        next_step=session.next_step(),
        blockers=session.blockers(),
        checkpoint_path=checkpoint.path,
    )
    return FailureReport(
        kind="budget_exhausted",
        message=f"Budget exhausted at step {session.current_step}. "
                f"State saved to {checkpoint.path}. Handoff at {handoff.path}. "
                f"Resume with: agent resume {handoff.path}",
        completed=session.completed_steps(),
        partial_artifacts=session.artifacts(),
    )
```

Budget exhaustion is **not a failure** — it is a planned stop. The budget exists precisely so that runaway sessions stop gracefully rather than producing five-figure bills. The harness handles it gracefully because it was designed to; an unplanned OOM kill mid-step is the failure mode the checkpoint prevents.

## Crash recovery

A harder case: the process is killed externally (OOM, SIGKILL, machine reboot) before the graceful-degradation path runs. The defense is Module 8's **atomic checkpoint on every meaningful step**: a crash leaves either the pre-step or post-step checkpoint, never a half-written one. On restart, the harness reads the latest checkpoint, verifies it is well-formed, and resumes from there. Side effects already applied (a commit, an email sent) are not re-applied — idempotency (7.2) is what makes this safe.

The recovery contract: **after a crash, the harness resumes from the last checkpoint with no duplicated side effects.** This is the property the Module 8 lab verifies by killing the process mid-task.

## Human-readable failure reports

When the agent cannot finish, the failure report must answer five questions:

1. **What was the task?** (Restate the goal, so the human does not have to re-derive it.)
2. **What was completed?** (List finished steps and artifacts produced.)
3. **What failed?** (Name the specific step and error.)
4. **Why did it fail?** (The root cause as the agent understands it, with evidence.)
5. **What should the human do next?** (The next action, or "this needs a human decision.")

Vague failures ("task failed") fail all five; specific failures ("could not resolve type error in auth.ts:42 — the Token type is imported from two conflicting modules, `@auth/core` and `@internal/auth`. Recommend consolidating on `@auth/core` and removing the alias in tsconfig paths") pass all five. The report is itself a prompt — to the human.

---

## Anti-Patterns

### Retry-without-jitter
Fixed-delay retries. Under load, every client retries at the same offset and re-overloads the service. Cure: full-jitter exponential backoff (7.2).

### Retry-without-idempotency
Retrying `send_email` or `charge_card` directly. Two emails sent; two charges hit. Cure: idempotency keys or "do not retry" classification for non-idempotent tools.

### The swallowing catch
`try { tool() } catch { return "ok" }`. The model gets no error and proceeds on a false success. Cure: structured `{ok, error, retryable, category}` returns, never silent recovery.

### The session-wide breaker that never resets
Tripping the circuit for the whole session on a transient blip. The tool stays disabled even after the downstream recovers. Cure: half-open probes after a cooldown for tools that can recover.

### The vague failure report
"Task failed." The human cannot act. Cure: the five-question report (7.4).

---

## Key Terms

| Term | Definition |
| --- | --- |
| **Error taxonomy** | The 4 categories: transient, LLM-recoverable, user-fixable, fatal |
| **Error compounding** | Per-step error rates multiply; 99% over 50 steps = 60% |
| **Exponential backoff + jitter** | Growing delay with random spread; prevents thundering herd |
| **Full jitter** | Uniform `[0, cap]` backoff; best under contention |
| **Idempotency key** | Client-supplied token that makes a non-idempotent tool safe to retry |
| **Stuck loop** | Same tool + same input + same error, repeated |
| **Circuit breaker** | Disable a failing tool after N consecutive failures, for the session |
| **Half-open breaker** | Re-allows a probe call after cooldown; recovers transient outages |
| **Graceful degradation** | Partial completion + clear failure report; not silent failure or full abort |

---

## Lab Exercise

See `07-lab-spec.md`. Write a stuck-loop detector as a harness middleware. Intentionally exhaust the budget at step 15 of a 30-step task. Compare failure behavior: does the harness save state and report clearly, or crash?

---

## References

1. **AWS Architecture Blog (2015)** — *Exponential Backoff and Jitter*. The canonical analysis of full-jitter vs. alternative backoff strategies under contention.
2. **Stripe API documentation** — idempotency keys; retry cap of 2.
3. **Hystrix Wiki (Netflix)** — the circuit-breaker pattern, including half-open recovery.
4. **Yao et al. (2022)** — *ReAct*. arXiv:2210.03629. The error-compounding motivation for ReAct loops (Module 1).
5. **Module 1.2** — the error-threshold stop condition and the infinite loop problem.
6. **Module 1.4** — the observability payload (`input_hash`, `output_hash`) that powers stuck-loop detection.
7. **Module 2.2** — the "never throw, always return a structured error" rule that makes classification possible.
8. **Module 4.2** — the handoff file for budget-exhaustion resume.
9. **Module 6.2** — alert fatigue; why user-fixable errors should interrupt, not all errors.
10. **Module 8** — checkpointing for budget-exhaustion and crash recovery.