DevOps — Frequently Asked Questions
Practical answers on CI/CD and operations — how often to deploy, trunk-based development, what to do about flaky tests, blue/green versus canary, on-call for small teams, and which metrics are worth tracking.
Delivery#
How often should we deploy?#
More often than you currently do, in smaller pieces. Deployment risk is roughly proportional to the size of the change, and batching changes to reduce the number of deploys concentrates the risk instead of removing it.
The counter-intuitive part: frequent deploys make each one boring. A team that ships daily has a rehearsed path, a fast rollback and small diffs to bisect. A team that ships quarterly has an event, and events go wrong.
Do we need feature branches or trunk-based development?#
Trunk-based, with short-lived branches measured in hours or a couple of days, works better for most teams. Long-lived branches accumulate merge risk that surfaces exactly when you are trying to release.
If a feature needs longer than that, hide it behind a flag and merge the incomplete work. Merging often is what makes integration cheap; the branch model is secondary.
What should the pipeline actually do?#
Build once, then promote the same artefact through environments. A separate build per environment breaks the chain of evidence between what was verified and what is now serving traffic.
Beyond that: fast feedback first (lint, unit tests, build) so failures come back in minutes; integration tests next; deploy to a staging environment automatically; and production deployment behind whatever approval your risk level genuinely requires — no more.
Our pipeline takes 40 minutes. Does it matter?#
Yes, and not linearly. Past roughly ten minutes people stop waiting for it, context-switch, and come back to a failure they have to reload into their heads. Past thirty, they start batching changes to avoid the wait, which reintroduces exactly the risk CI was meant to remove.
Split fast checks from slow ones, run in parallel, cache dependencies properly, and be willing to move the slowest suite to a scheduled run rather than blocking every commit on it.
Testing in the pipeline#
What do we do about flaky tests?#
Quarantine them immediately, then fix or delete them on a deadline. A suite with known flakes trains everyone to re-run red builds, which means the suite no longer blocks anything — you are paying for the runtime and receiving no signal.
Track the flake rate as a first-class metric. It is one of the clearest indicators of whether a team trusts its own pipeline.
Should a failing test block a deploy?#
Yes, or delete the test. A test that can be ignored is documentation with a runtime cost.
The pressure valve people actually need is a documented emergency path — an override that requires a named approver and produces a record. Used twice a year that is healthy; used weekly it means the suite does not reflect what the team believes.
Releasing safely#
Blue/green or canary?#
Canary when you have enough traffic for a small percentage to be statistically meaningful, and metrics good enough to compare the two populations. It catches problems that only appear under real traffic while exposing few users.
Blue/green when traffic is low or the deployment unit is hard to split. It gives a clean, fast switch back, at the cost of running two environments and dealing with shared state — the database is usually the complication in both cases.
For many small services, a rolling deploy with a fast rollback is sufficient and much simpler. Complexity in release strategy should be paid for by a risk you can name.
How do we handle database migrations safely?#
Expand, then contract. Add the new column or table, deploy code that writes both old and new, backfill, switch reads, and only then remove the old shape — several deploys apart.
The rule underneath: every migration should be backward compatible with the currently running code. Otherwise deploy and rollback both become outages, and you discover this at the worst moment.
When is a change freeze justified?#
Around a genuinely high-stakes business period, briefly. As a general practice it backfires: work accumulates behind the freeze and lands in one large batch immediately afterwards, which is the riskiest deploy of the quarter.
If your team needs a freeze to feel safe, the underlying problem is confidence in rollback, not frequency of change.
Operations#
How do we run on-call with a small team?#
Honestly. Decide what genuinely justifies waking someone — customer-affecting and actionable — and route everything else to working hours. Every alert that pages without requiring action erodes the response to the ones that do.
Write runbooks for the top few failures so the person paged does not have to think from first principles at 03:00. Rotate fairly, count the hours, and act on the reviews. On-call that nobody improves is how teams lose their best operators.
What should we monitor?#
Start with what tells you a customer is affected: error rate and latency at the entry point, then saturation of whatever runs out first. Add a check that the thing is actually processing work, not merely responding to health probes — a service can be perfectly healthy and doing nothing.
Then be strict about the split: alerts for what needs a human now, dashboards for everything else.
Which metrics are worth tracking for the team itself?#
The four that have held up across a lot of research and practice: deployment frequency, lead time from commit to production, change failure rate, and time to restore service.
Use them as a conversation about the system, not as individual performance measures. The moment they become targets for people rather than signals about the delivery process, they get gamed and stop being informative.
Is infrastructure as code worth it for a small setup?#
Yes, mostly for a reason people underrate: it is the only reliable documentation of what exists. Manually configured infrastructure is understood by whoever built it, until they leave.
Start with the pieces that would be painful to rebuild, keep the state file safe and backed up, and accept that some things will be clicked in the console early on — just write down which ones, so the gap is known rather than assumed away.