Worked Example — Twelve Services, One Deployment

This is an illustrative example. The company, migration and figures are invented. The outcome — a distributed system with all the costs of distribution and none of the independence — is the most common way this migration goes wrong.

The situation#

A retail company split a monolith into twelve services over eighteen months. The stated goals were reasonable and were written down: independent deployment, independent scaling, and teams able to work without coordinating.

Eighteen months later, none of them had been achieved.

Every measure was worse. The team had not done the work badly; they had done a great deal of work carefully, and it had produced a distributed monolith.

Why they could not deploy separately#

Three causes, each of which independently prevents independent deployment.

All twelve shared one database schema. Splitting the code had been treated as the migration; the data had been left. Any change to a table potentially affected several services, so schema changes required coordinating everyone — which is precisely the coordination the split was meant to remove.

Interfaces between services were not versioned. A change to a response shape broke the consumers immediately, so producer and consumers had to deploy together. Twelve services with unversioned interfaces form one deployment unit with extra network calls.

A shared library held the domain model, used by all twelve. Changing it meant rebuilding and redeploying all twelve, and the library changed most weeks because it held exactly the things that change.

Any one of these makes independent deployment impossible. All three were present.

Why it got slower#

Checkout had been one process making local function calls. It became seven synchronous network calls in sequence.

Worse than the median is the tail behaviour. Seven sequential dependencies mean seven chances of a slow response, and the slowest determines the total. The p95 got much worse than the median because a request now needs all seven to be fast rather than one process to be fast.

Availability moved the same way. Seven dependencies at 99.9% give checkout a ceiling of 99.3% before checkout's own code does anything wrong.

Why incidents took longer#

Diagnosis went from reading one log to correlating twelve, with no shared request identifier. The migration had not included distributed tracing, on the reasonable-sounding grounds that it could be added later.

Later arrived as a 2.5-hour median diagnosis time, and it arrived during incidents.

What was actually fixed#

Not a return to the monolith. Eight months of work, in a specific order.

Tracing first. A request identifier propagated through every call, and traces collected centrally. Three weeks, and it cut median diagnosis from 2.5 hours to 25 minutes before anything structural changed. It also produced the latency breakdown above, which had been guesswork.

The shared library was broken up. Each service got its own model of the concepts it uses. This felt wrong to several engineers — the same concept now defined in several places, which reads as duplication. It is the point: two services that must agree on a model must deploy together, and the shared library was the mechanism enforcing that. Some duplication is the price of independence, and it is cheaper than the coordination it replaces.

Interfaces were versioned, with a rule: a producer supports the previous version for one release. Consumers can now upgrade on their own schedule, which is what independent deployment means in practice.

The schema was split, service by service, over four months. Each service owns its tables. Where another service needs the data, it asks, or receives an event. This was the largest piece of work and the one that actually delivered the goal.

Checkout's seven sequential calls became three, with two of the remaining moved off the request path entirely — inventory reservation and loyalty accrual now happen asynchronously, because neither needs to complete before the customer sees a confirmation.

Four services were merged back into two. They were always changed together, always deployed together, and owned by the same team. The boundaries had been drawn along technical layers rather than along the business, which is what produces services that cannot move independently.

The result#

The migration took eighteen months and delivered nothing. Making it work took eight more. Checkout latency remains above the monolith's, which is the honest cost of distribution and is acceptable at 310 ms.

What was learned#

Splitting the code is not the migration. Data, interfaces and shared libraries each independently prevent independent deployment, and all three are easier to leave alone than to address.

Some duplication is the price of independence. The shared library felt like good practice and was the mechanism forcing twelve services to move together.

Draw boundaries along the business, not along layers. The four services merged back had been split by technical concern, so they always changed together.

Add tracing before you split, not after. Three weeks of work cut diagnosis time by 80% and supplied the measurements every later decision depended on. It was deferred because it delivers nothing visible, and it was the highest-value item in the whole programme.

Back to Architecture