Rollback Plan Checklist

Record the currently-deployed git sha, container image tag, and Helm chart version so the rollback target is unambiguous. Confirm the previous image is still present in ECR/GCR and has not been pruned by registry lifecycle rules — a common gotcha when the rollback target is more than 30 days old.

Review any DB migrations in this release for reversibility. Destructive ops (DROP COLUMN, DROP TABLE, NOT NULL backfills) are not safely reversible by re-running the prior image — flag these for expand/contract treatment. If the migration is one-way, the rollback plan must rely on forward-fix, not down-migration.

Name the release captain (drives the deploy), the incident commander if rollback fires, and the comms lead who owns the status page and #incidents updates. Both primary and secondary on-call should be reachable for the duration of the deploy window.

Take a manual RDS / Cloud SQL snapshot tagged with the release candidate version. Don't rely on the nightly automated snapshot — it may be 16 hours stale by deploy time. Note the snapshot identifier in the deploy ticket.

Restore into a non-prod instance and run a smoke query against a known-row-count table. A backup that has never been restored is not a backup. Common failure: the restore script depends on a credential that rotated last quarter.

Pin the four golden signals: request rate, error rate, p95 / p99 latency, and saturation (CPU, memory, DB connections). Layer the deploy marker on the dashboard so the canary's effect is visually unambiguous.

Pre-commit to numeric criteria: e.g., error rate > 2% sustained 5 min, p99 latency > 1.5x baseline for 10 min, or any SEV2 customer report tied to the release. Decide before deploy, not during — debating thresholds at 3am is a known anti-pattern.

If the release shipped behind a kill-switch flag, flip it off first — it's the fastest path to mitigation and avoids a full redeploy. Only proceed to image revert if the flag doesn't fully gate the bad code path.

Shift the load balancer / service mesh weights from the canary back to 0%. Watch error rate trend down as traffic drains. If using ArgoCD rollouts, abort the rollout rather than rolling forward.

Run the rollback pipeline pointing at the prior image tag captured in pre-release. For Kubernetes, kubectl rollout undo on each affected Deployment, or set the chart values back and re-sync ArgoCD. Confirm pod readiness gates pass before declaring the rollback complete.

Only run the down migration if pre-release confirmed it as reversible. For one-way migrations, the prior image must be forward-compatible with the new schema; if not, restore from the pre-deploy snapshot — but only with the engineering director's explicit approval since this drops any writes since the snapshot.

Update Statuspage / Better Stack with the user-visible impact, time window, and current state (investigating → identified → monitoring → resolved). Notify customer support so inbound tickets are routed correctly.

Invite the release captain, on-call, IC, comms lead, and at least one engineer not on the team. Blameless framing — focus on contributing factors (alert tuning, missing canary check, untested migration), not individuals. First-shift narratives are usually wrong; the PIR is where the real causes surface.

Walk the timeline backwards. Look for second-order causes: was the canary cohort too small to surface the issue? Did the dashboard hide the regression? Was the test that should have caught this missing or flaky-and-ignored? Capture each contributing factor as a discrete action item with a named owner.

Each action item gets an owner, a target sprint, and a link back to the PIR. Common gap: action items filed but never tracked to closure — six months later the same incident recurs. The engineering manager owns chasing these to done.