# Chaos Experiment Report — Sample

**This is an illustrative example.** The system, experiments and figures are invented. The
method is what transfers: state a hypothesis, break something on purpose, and record where you
were wrong.

---

## Chaos experiments — AI assistant dependencies

| | |
|---|---|
| System | Customer assistant: retrieval, model provider, account service, cache |
| Period | 2026-07-14 to 2026-07-17 |
| Environment | Production, 5% of traffic, during working hours with the team present |
| Abort criteria | Error rate above 2% on the affected slice, or any data written incorrectly |
| Experiments | 6 |

## 1. Summary

| # | Experiment | Hypothesis held? |
|---|---|---|
| 1 | Model provider returns 503 | **No** |
| 2 | Model provider latency ×5 | **No** |
| 3 | Retrieval returns empty | Yes |
| 4 | Account service unavailable | **No** |
| 5 | Cache unavailable | Yes |
| 6 | Model provider returns malformed output | **No** |

**Four of six hypotheses were wrong**, which is a good outcome for an exercise whose purpose
is finding out. An experiment that confirms what you assumed has told you very little.

## 2. Experiment 1 — provider returns 503

**Hypothesis.** The assistant shows a clear error and the user can retry. Nothing else is
affected.

**What happened.** The client retried three times with no backoff, then the request timed out
after 45 seconds. The user saw a spinner for the full 45 seconds followed by a generic error.
Worse, the retry storm from a few hundred concurrent users **saturated the connection pool
shared with the account service**, so order lookups — which need no model at all — began
failing too.

**A failure in one dependency took out a feature that does not use it.** That coupling was
invisible in every architecture diagram, because the diagrams show services and the shared
pool is configuration.

**Fixed.** Exponential backoff with jitter, 3 attempts, 8-second total budget. Separate
connection pools per downstream. Clear message at the end of the budget.

## 3. Experiment 2 — provider latency ×5

**Hypothesis.** Responses are slow. Users wait. Nothing breaks.

**What happened.** Requests exceeded the 30-second client timeout, so users retried manually.
Each manual retry started a new request while the original continued and completed — and was
charged. Model spend on the affected slice rose 3.4× for a period in which **no user received
a successful answer**.

The system had no concept of a request being abandoned. Nothing cancelled the upstream call
when the client went away.

**Fixed.** Cancellation propagates on client disconnect. A per-user in-flight limit prevents a
retry storm from one person.

## 4. Experiment 4 — account service unavailable

**Hypothesis.** The assistant answers general questions and declines account-specific ones.

**What happened.** It answered account-specific questions anyway — **from stale cached data
with no indication it was stale**. One test user was shown an order status that had changed
two days earlier.

This is the most serious finding of the four days. The fallback was working exactly as
designed, and the design was wrong: a silent fallback to old data is worse than an error,
because the user acts on it.

**Fixed.** Cached account data older than 15 minutes is served with a visible timestamp and a
notice. Anything older than an hour is not served at all; the assistant declines and says why.

## 5. Experiment 6 — malformed model output

**Hypothesis.** The parser rejects it and the request fails cleanly.

**What happened.** The parser was lenient. Given truncated JSON it recovered what it could and
passed a **partial object** downstream — a summary with the caveats section missing, and no
indication that anything was absent.

A strict parser would have failed the request. The lenient one produced something that looked
complete.

**Fixed.** Schema validation with no recovery. Malformed output fails the request, is logged
with the raw response, and retries once.

## 6. What held

**Experiment 3 — retrieval returns empty.** The assistant correctly said it could not find
relevant information and did not attempt to answer from general knowledge. This is the
behaviour teams most often get wrong, and it was right here because it was explicitly designed
and tested.

**Experiment 5 — cache unavailable.** Latency rose about 40%, everything remained correct, no
errors. The cache is genuinely a cache.

## 7. The pattern

Every failure had the same shape: **the system degraded into something that looked like
success.**

A retry storm looked like slowness. Stale data looked like data. Partial output looked like
output. In each case the failing dependency was handled by code written to be helpful, and
being helpful meant hiding the failure from the person who needed to know about it.

The three experiments that behaved well were the ones where somebody had explicitly decided
what failure should look like.

## 8. Actions

| Action | Status |
|---|---|
| Backoff and a total time budget on model calls | Done |
| Separate connection pools per downstream | Done |
| Cancellation on client disconnect | Done |
| Stale account data: visible timestamp, hard limit at 1 hour | Done |
| Strict schema validation on model output | Done |
| Per-user in-flight request limit | Planned |
| Re-run all 6 experiments monthly | Scheduled |

## 9. Method notes

Run in production on 5% of traffic, during working hours, with the team watching and abort
criteria agreed in advance. Staging would not have found experiment 1 — the shared connection
pool exists only in the production configuration.

Customer support knew the window and had a script. Two tickets arrived during experiment 2 and
both were resolved by explaining that a test was in progress.

---

## Notes on using this format

**Write the hypothesis before the experiment.** Four of six were wrong, and that is the value.
Without a written hypothesis, the result is just an incident you caused.

**Break things in production, carefully.** Small slice, working hours, team present, abort
criteria agreed. The most serious finding existed only in the production configuration.

**Look for degradation that resembles success.** Every finding here was a fallback doing its
job. Silent recovery is the failure mode chaos experiments are best at exposing.

**Re-run on a schedule.** These fixes will decay, and a new dependency added next quarter will
arrive with its own optimistic fallback.