Deployment Checklist

Branch from main and push a release-candidate tag (e.g., v2024.45.0-rc.1) following semver. The release captain owns this step. Make sure no late-merging PRs slip in after the cut without re-tagging.

Cross-check PRs merged since the last release tag against the changelog. Flag breaking changes, deprecations, and customer-visible behavior so support and docs aren't surprised on release day.

Deploy the RC to staging and run Playwright/Cypress e2e + integration tests. Do not ignore flakes — re-run once, then triage any persistent red. Required status checks must be green before sign-off.

QA exercises sign-up, login, billing, and the top-three customer workflows in staging. Capture any regressions in Linear/Jira tickets linked to the release ticket.

Verify the previous container image is still in the registry (not pruned), the DB migration is reversible or has no destructive change, and the rollback runbook references the correct sha. Quarterly rollback drill is what proves this works — release-day verification just confirms artifacts exist.

On large tables, ADD COLUMN with a default, ALTER COLUMN, or non-concurrent index creation can take exclusive locks for hours. Use CREATE INDEX CONCURRENTLY, split adds from defaults, and batch backfills with sleeps to keep replication lag bounded.

Restore last night's prod backup into the staging DB and run the migration end-to-end. Record duration and peak replication lag so the on-call knows what to expect during the real run.

Check PagerDuty and the incidents channel. Don't deploy on top of an active incident — you'll obscure the contributing factor and complicate rollback.

Both the release captain and the primary on-call should be at a keyboard for the duration of the deploy plus the 30-minute monitoring window. No deploys with the on-call boarding a flight.

Apply the migration before the backend deploy so the new code lands on a compatible schema. Watch replication lag in Datadog/Grafana — pause if lag climbs past the SLO.

Route 5% of traffic via the load balancer or service mesh weight. Watch error rate, p99 latency, and saturation for 10 minutes against the golden-signals dashboard.

Step the weight up gradually with a watch interval at each stage. If the error rate or latency dashboards show regression at any step, hold and investigate before proceeding.

Frontend ships after backend is at 100% — backend is forward-compatible, frontend assumes the new API. Invalidate CDN caches per the release runbook.

Open an incident in PagerDuty/Incident.io, name an IC, and post to #incidents. Don't try to debug forward — get back to the last known-good state first, then investigate.

Use the previous image tag captured pre-deploy. If a migration ran, follow the documented down-migration or use the expand-migrate-contract fallback to keep the old code compatible with the new schema.

Stay on the golden-signals dashboard — latency, traffic, errors, saturation. Sentry / Bugsnag spikes by release tag are the fastest signal of a regression that synthetic tests didn't catch.

Only flip flags listed in the release plan, one at a time, with a few minutes between each so you can attribute any regression to the specific flip. Note flag owner so cleanup happens in the next quarterly flag review.

Promote the RC tag to the GA version (e.g., v2024.45.0). Push the tag and confirm the release shows up in GitHub Releases / GitLab Releases.

Customer-visible release notes go to the public changelog or status page. Internal-only details (infra changes, refactors) stay in the engineering changelog.

Hotfix candidates, runbook gaps, dashboard gaps, alert tuning — log them in Linear/Jira while the context is fresh. These become the action items if a retro is scheduled.