Worked Example — The Quarter We Stopped Exploratory Testing
A worked example of a team that automated everything and stopped looking — what escaped, why the automated suite could not have caught it, and what four hours a week restored.
This is an illustrative example. The team, quarter and figures are invented. The reasoning that led to it — full automation, therefore manual testing is redundant — is sound-sounding and wrong in a specific, measurable way.
The situation#
A team with a mature automated suite: 3,100 tests, 91% line coverage, running on every change in 12 minutes. It was genuinely good work and had taken two years.
Alongside it, two testers ran exploratory sessions — roughly eight hours a week between them, charter-based, unscripted.
Under delivery pressure in April, the sessions were dropped. The reasoning was explicit and was written in the planning notes: "coverage is 91% and everything is automated; exploratory testing is duplicating what the suite already does."
The automated suite continued to pass. Coverage stayed at 91%. Nothing appeared to change.
What escaped over the quarter#
| Q1 (with sessions) | Q2 (without) | |
|---|---|---|
| Defects found before release | 47 | 21 |
| Defects found by customers | 6 | 23 |
| Customer-reported, severity high or above | 1 | 7 |
| Automated suite pass rate | 100% | 100% |
| Coverage | 91% | 91% |
Total defects were roughly the same. What changed was who found them. Twenty-six defects moved from being found internally to being found by customers, and seven of those were serious.
Every one of them shipped through a suite that passed completely.
The seven serious ones#
Examined individually, because the pattern is the point.
Two roles could each perform an action the other could not, inverted. The permission matrix was tested; each role was tested against each action; both tests passed because the expectations had been written from the same misunderstanding as the code.
Bulk operations timed out at customer scale. Tested at 500 records. One customer routinely does 40,000.
Two features interacted. Scheduling a report and changing a template at the same time produced a report with mixed formatting. Both features were fully tested. Nothing tested them together, because nobody had thought of the combination.
An expired session mid-workflow lost twenty minutes of entered data with no warning. Every individual screen handled expiry correctly; the multi-step flow did not.
A currency field accepted a negative value in one path of three. The other two validated.
The export encoding broke on non-Latin characters for one customer whose data had always been in English until it was not.
A confirmation dialogue appeared behind a modal in one browser at one window size, making the action impossible to complete.
Not one of these is a coverage gap. Every line involved was covered. The defects live in combinations, scales, timings and states that no test asserted because no one imagined them. That is not a criticism of the suite — it is what an automated suite is.
What was reinstated#
Four hours a week, not eight, and structured differently.
Charter-based sessions. Each session has a stated mission — "use the refund flow in every state a payment can be in", "work with the export while another user changes the data", "complete this workflow as every role, including one whose session expires". Not a script; a boundary.
Two hours per session, notes taken during, findings written up after.
Charters chosen from where risk is, and from where the last escape came from. The interaction between features became a standing charter because that category produced three of the seven.
Everything found becomes an automated test. The exploratory session finds it once; the suite then catches it forever. This is the division of labour that makes both worth having — sessions find the unimagined, automation remembers it.
The result#
| Q2 (none) | Q3 (4 hrs/week) | |
|---|---|---|
| Defects found before release | 21 | 39 |
| Defects found by customers | 23 | 8 |
| High severity, customer-found | 7 | 1 |
| Automated tests added from session findings | — | 31 |
| Cost | 0 | ~52 hours |
Fifty-two hours of testing moved 22 defects from customers to the team, and added 31 permanent tests the suite would never have contained.
The comparison the team now uses internally: one customer-found high-severity defect consumes, on average, about 30 hours across support, investigation, fix, release and the follow-up conversation. Six of them cost more than the entire quarter of sessions.
What was learned#
Coverage measures which lines ran, not which behaviours were checked. Ninety-one per cent was true throughout, and it was true during the quarter that produced seven serious escapes.
An automated suite tests what somebody imagined. That is its nature. It cannot find the combination nobody considered, the scale nobody tried, or the timing nobody thought about.
The failures cluster in combinations, scale, timing and state. Every one of the seven was in a gap between things that were individually well tested.
Sessions and automation are not alternatives. Sessions find things once; automation remembers them forever. Dropping either leaves the other doing a job it is not shaped for.