Bad deploys undo themselves.
About eight in ten production incidents start with a change. Right now, when a deploy breaks something, a person has to notice, find the last good version, and push it back. In two weeks, your five most important services will send new versions to a small slice of traffic first, watch the metrics, and roll back on their own in under two minutes if anything gets worse.
Two weeks. Five services. Fixed price. You watch it roll back a real bad deploy in production before we invoice.
- 14:02:10Deploy marker posted. v2.14.0 receives 5% of traffic. Analysis starts against v2.13.2 as the stable baseline.
- 14:03:14Analysis check 2 of 2 failed: canary 5xx rate 3.1× stable for two consecutive intervals. Threshold is 1.5×.
- 14:03:44Rollout aborted. 100% of traffic back on v2.13.2. Bad version kept at zero traffic for inspection. Rollback marker posted, Slack notified.
One tool, one outcome, one demo, one price.
This is deliberately the smallest thing that changes how deploys fail. No SLO workshop, no policy engine, no new platform. Argo Rollouts on the five services that matter, tuned against your real traffic, and proven in your production.
New versions get a slice of traffic first
Each of the five services deploys through a canary: 5% of traffic, then 25%, then 50%, then all of it. The stable version keeps serving the rest and is never touched until the new one has earned it.
Works with the ingress controller, Gateway API, or service mesh you already have. No mesh required.
The metrics decide, not a person watching a dashboard
At every step, error rate and latency on the canary are compared to the stable version, every 30 seconds, using the metrics backend you already run. Thresholds are tuned from a week of the service's real traffic, not guessed.
Prometheus, Datadog, New Relic, or CloudWatch. A business metric like checkout completions can be added if you already emit one.
Regression means the old version is back in seconds
If the canary fails two consecutive checks, the rollout aborts and all traffic returns to the stable pods, which were running the whole time. The bad version stays deployed at zero traffic so your engineers can inspect it. Nobody gets paged for a deploy that already fixed itself.
Rollback is exercised deliberately in staging for every service, and once in production for the acceptance test.
Every deploy and rollback shows up on your graphs
The pipeline posts a marker to your dashboards for each deploy and each rollback. The first twenty minutes of every incident, spent asking "what changed?", go away.
Grafana annotations or the Datadog events API, from whichever pipeline you run. Argo CD, Flux, GitHub Actions, GitLab CI, Jenkins.
Two days to scope, ten days to done.
About ten hours of your team's time across the two weeks. Normal deploys continue throughout. The only thing engineers notice is that a deploy takes a few minutes longer to finish, and that they've stopped watching graphs afterward.
Pick the five, confirm the fit
We read the cluster, the ingress setup, the metrics backend, and the pipeline. We pick the five services with you and write a one-page work order with a fixed price.
- Preconditions checked (see below)
- Five services chosen together
- Four done-when lines, fixed price
- Your go / no-go
Install, tune, prove in staging
Argo Rollouts in the cluster, deploy markers wired from your pipeline, and each service converted to a canary with an analysis template tuned to its real traffic. Every service is proven in staging with a deliberate failure before it moves.
- Pull requests into your repositories
- One analysis template per service tier, reusable
- Rollback demonstrated in staging, twice
- Working demo on day 4 and day 7
Production, then the test that counts
All five services move to canary deploys in production during a window you choose, with us on-call. The next day we deploy a deliberately broken build of one service to production and watch it roll back on its own, with your engineer at the keyboard.
- Cutover in your window, us on-call
- Deliberate bad deploy, rolled back automatically
- Under two minutes, zero human actions
- You sign off on the test, not a report
Your team adds service six
A runbook for adding the next service, a recorded walkthrough, and a pairing session where your engineer converts a sixth service with us watching. Then 30 days of support.
- Runbook and notes in your wiki
- Recorded walkthrough
- Pairing session: service six, your hands
- 30 days of post-handover support
Four done-when lines. You sign off on the fourth.
The work order is a page long. Done means the acceptance test passed in your production while your engineer watched, not that a document was delivered.
- Canary deploys on five servicesA small traffic slice first, automated analysis, then promotion. In production.
- Automatic rollback under two minutesA regression in error rate or latency on any of the five rolls back with no human action.
- Deploy and rollback markersEvery deploy and rollback on the five services appears on your dashboards.
- The acceptance test, recordedA deliberately broken build deployed to production and rolled back automatically, with timestamps. Yours to keep.
- Reusable analysis templatesOne per service tier, tuned to your traffic, so adding service six is a copy and a label.
- Runbook, walkthrough, pairing, supportYour team owns it. 30 days of support after handover.
- Scoping is two days at $1,500, credited in full against the project.
- Half on signature of the work order, half when you sign off on the acceptance test.
- Additional cluster or environment: $4,000. Each further block of five services: $6,000.
- If it takes us longer than ten days, that's on us, not your invoice.
Your team ships faster than it did a year ago. The safety net didn't grow with it.
Nearly every engineering team adopted AI coding assistants in the last eighteen months. More pull requests, larger diffs, faster merges, and reviewers reading code nobody on the team wrote. Google's DORA research found the result: throughput up, delivery stability down.
The old safety net was an engineer watching a dashboard for ten minutes after each deploy. That worked at five deploys a day. It doesn't work at twenty. Amazon learned this in March 2026 and spent 90 days rebuilding change safety across its critical systems. A hundred-person company can't afford a 90-day reset, and can't afford the outage either.
This product is the smallest piece of that safety net, sized for a team your size, done in two weeks.
Five preconditions. We check them on the scoping call.
If any of these is a no, this product isn't the right first step, and we'll say which project is. Usually it's the observability work, so the canary has metrics to judge by.
Good fit
- Kubernetes in production, with the five services running as Deployments.
- Traffic through an ingress controller, Gateway API, or a service mesh. NGINX, Traefik, AWS Load Balancer Controller, Istio, and Linkerd all work.
- A metrics backend that already records request rate, error rate, and latency per service: Prometheus, Datadog, New Relic, or CloudWatch.
- Deploys go through a pipeline or GitOps tool we can add a step to.
- Someone who can approve a deliberate bad deploy to production, in business hours, for the acceptance test.
Not this product
- Databases, queues, and other StatefulSets. Argo Rollouts doesn't cover them, and deploy-caused incidents come from application services anyway.
- SLOs, error budgets, and deploy gating. That's the next project, once you've seen this one work. See contract engineering.
- Config and infrastructure changes. Different mechanism, different project.
- No per-service error rate or latency metrics yet. The canary has nothing to judge by. Start with observability.
The things people ask on the scoping call.
We already have Argo Rollouts installed.
Won't canaries slow our deploys down?
Our services might not handle two versions running at once.
What about our databases?
Do we need a service mesh?
We could do this ourselves.
What happens after?
Tell us how long your last bad deploy lived in production.
A 30-minute scoping call covers your stack, your five services, and the preconditions. If this isn't the right first project, we'll say which one is.