Performance Testing AI Systems: Latency, Throughput and Cost Per Answer

Performance testing an AI feature differs from conventional load testing in one decisive way: you usually do not control the bottleneck. The model runs on someone else's infrastructure, behind a rate limit, with latency you cannot tune.

That changes what you measure and what you can do about it.

What to measure#

Cost belongs in this table. In conventional systems performance and cost are loosely coupled; with per-token pricing they are the same variable. A change that doubles context length degrades latency and doubles the bill.

The tests#

Baseline. Single request, no load, repeated. Establishes best-case latency and its variance. Variance matters: if p50 is 1.2s and p99 is 9s at zero load, that spread is inherent and will only widen.

Sustained load. Expected concurrency for a sustained period. Watch for rate-limit rejections and rising queue depth — those appear before latency does.

Spike. Sudden burst. Does it queue, shed, or fail? Decide which of those you want before the test tells you.

Long-input. Requests near the context limit. Latency scales with input length, and behaviour often degrades quietly near the limit.

Degraded-backend. Simulate the provider being slow or returning errors. This is the test that matters most, because it is the failure you will actually experience and cannot prevent.

Cost-under-load. Run the sustained test and measure spend. Retries multiply cost invisibly.

The failure modes specific to AI#

Rate limits, not CPU. Your constraint is a quota. Load testing that saturates your own servers tells you nothing — the queue in front of the provider is the system.

Retries multiplying cost. A timeout triggers a retry, the retry also runs, and you pay for both. Under load this compounds. Measure cost per completed request or you will not see it.

Context growth. Longer prompts are slower and more expensive. A feature that adds "just a bit more context" changes the performance profile of every request.

Cold starts on self-hosted models — first request after idle can be dramatically slower.

Streaming that masks a problem. Time to first token stays flat while total time degrades. Users notice later, and your dashboard says everything is fine.

What you can actually change#

Since you do not control the model:

Cache. Identical or near-identical requests should not be recomputed. At volume this is the single largest lever on both latency and cost.

Shorten the prompt. Directly reduces both. Track token count as a metric — if a prompt grows 40% and quality does not move, that is a regression.

Retrieve fewer chunks. More context is not reliably better, and beyond a point it dilutes the answer as well as slowing it.

Use a smaller model where it passes. Model pricing spans a thousand-fold range; quality on well-specified tasks spans far less. See Model Evaluation.

Stream. Does not reduce total time; transforms the experience.

Queue explicitly. An honest queue with a position indicator beats an unbounded wait.

Degrade deliberately. Decide in advance what happens at capacity — a cheaper model, a cached answer, or a clear "try again shortly". The default is a timeout, which is the worst option.

Setting a target#

Work backwards from the interaction:

Set targets on p95, not the average. And set a cost ceiling per request at the same time — without one, the natural response to every quality problem is more context, and nobody notices the bill until it is a line item.

FAQ#

How do we load test without a large bill?#

Test against a cheaper model to validate the harness, then run a short burst against the real one. You are measuring the shape of the curve; you do not need hours of it.

Should we test the provider's reliability?#

Test your response to their unreliability. You cannot fix their outage; you can decide whether your system queues, degrades or fails clearly. That is the testable part.

Is average latency ever useful?#

For capacity planning, mildly. For user experience, no — the average hides the tail, and the tail is what generates complaints. Report p50, p95 and p99.

How do we performance-test an agent?#

Steps per task multiply everything. Measure end-to-end completion time and cost per completed task, and watch step count — a task that took four steps last week and twelve today has degraded in performance even if it still succeeds.

What is the most common surprise?#

Cost, not latency. Teams size their infrastructure carefully and discover that retries, prompt growth and context expansion have tripled the per-request bill while every latency dashboard stayed green.

What else is coming for Performance Testing