Software Testing — Frequently Asked Questions
Practical answers on testing — how much is enough, what coverage really tells you, flaky tests, unit versus end-to-end, manual testing's place, testing in production, and who owns quality.
How much, and what kind#
How much testing is enough?#
Enough that you would deploy on a Friday afternoon. That is not a slogan — it is a usable threshold, because it forces you to ask what you are actually afraid of and whether your tests address it.
Concentrate effort where a defect would be expensive and where the logic is intricate. Spreading testing evenly across a codebase spends the same effort for less protection, because risk is never evenly distributed.
What is the right mix of unit, integration and end-to-end tests?#
Many fast tests close to the code, fewer tests that check components working together, and a small number of end-to-end tests covering the journeys that matter most.
The shape follows from cost, not doctrine: end-to-end tests catch integration problems nothing else will, and they are slow, more likely to be flaky, and painful to diagnose. Keep the ones that earn it — the handful of journeys where failure is unacceptable — and push everything else down to a faster level.
Is test coverage a useful measure?#
As a diagnostic, yes: coverage shows you which parts of the system no test touches at all, and that is worth knowing. As a target, no. Coverage rises when code is executed, not when behaviour is verified, so a coverage target reliably produces tests that execute a lot and assert almost nothing.
Better question: if this behaviour broke, which test would fail? If the answer is "none", write that test regardless of what the percentage says.
What should we test first in a system with no tests?#
The paths where failure is most expensive, and the areas that change most often. Then, whenever you fix a bug, write the test that would have caught it — that single habit builds a suite shaped by your actual failure history rather than by guesswork.
Do not attempt to retrofit coverage across the whole codebase. It stalls, and it produces tests nobody trusts.
Reliability of the suite#
What do we do about flaky tests?#
Quarantine them the day they are noticed, then fix or delete them on a deadline. A suite with known flakes trains the team to re-run red builds, and at that point the suite blocks nothing — you are paying the runtime and getting no signal.
Track the flake rate. It is one of the clearest available indicators of whether a team trusts its own tests, and it declines only when someone owns it.
Why are our end-to-end tests so unreliable?#
Usually timing and shared state. Tests that wait a fixed number of seconds rather than for a condition fail whenever the environment is slower than usual. Tests that depend on data left by earlier tests fail whenever order changes or one is run alone.
Make each test set up its own data and clean up after itself, wait on conditions rather than clocks, and make sure a test can be run in isolation. Most flakiness is one of those three.
Should a failing test block a release?#
Yes — or delete the test. A test that can be waved through is documentation with a runtime cost, and the team learns quickly which category each test is in.
What people genuinely need is a documented override with a named approver and a record. Used twice a year that is healthy judgement; used weekly it means the suite no longer reflects what the team believes about quality.
Manual and exploratory testing#
Is manual testing obsolete?#
Scripted manual regression testing mostly is — it is slow, tedious, and exactly what automation is good at. Exploratory testing is not, and it never will be.
Automation checks what someone already thought of. Exploratory testing finds the case nobody imagined: the unusual sequence, the odd input, the workflow a real user has that the team never considered. Those are consistently the more interesting defects, and they are found by a curious person with time and system knowledge.
Who should do the testing?#
Engineers own the tests around their own code — quality is not something that can be inspected in at the end by someone else. Specialists add most where the work is genuinely different: exploratory testing, designing the strategy, performance, accessibility, and asking the awkward questions early.
The arrangement to avoid is a separate team who receive finished work and are expected to make it good. That places responsibility away from the people who can act on it, and it makes the feedback loop as long as possible.
Environments and data#
Why do defects appear in production that passed in test?#
Because the environments differ, and usually in a way somebody knew about: smaller data volumes, stubbed dependencies, different configuration, a single instance instead of several, cleaner data, no concurrent load.
List the differences explicitly in your test plan. It converts an invisible assumption into a known limitation, and it tends to prompt someone to close the gap that matters.
Can we use production data for testing?#
Only masked or synthesised, with a documented basis. Copying live personal data into a test environment — where access is broader, retention is unmanaged and backups are casual — is a common and serious exposure.
Do borrow the shape of production data, though. Test data that is too tidy is a leading cause of production defects, because real data contains awkward names, unusual characters, historical records that predate current rules and volumes that expose slow queries.
Should we test in production?#
Some things can only be tested there — real traffic, real data volumes, real integrations. Feature flags, canary releases and synthetic monitoring are all forms of it, and they are legitimate.
The requirements are that it is deliberate, limited in blast radius, observable, and reversible. "Testing in production" as a description of having no other environment is a different thing entirely.
Process#
When should we write tests — before or after the code?#
Writing the test first tends to produce better-shaped interfaces and guarantees the test actually fails without the change, which is the property that matters most. Writing tests afterwards is better than not writing them.
The one non-negotiable: run the test against the unfixed code at least once. A test that has never failed has not been shown to test anything.
How do we know our testing is improving?#
Track defects found after release, how long they take to detect, the flake rate, and how long the suite takes to run. Those four move together with real quality.
Do not track tests written or coverage percentage as goals. Both are trivially increased without improving anything, and doing so is the path of least resistance when a number becomes a target.