Regression Testing Checklist

Provision the staging cluster from the same Terraform / Helm config used for production. Drift between staging and prod is the most common reason a green regression run still ships a regression — pin instance sizes, AZ layout, and k8s versions to match.

Confirm versions of upstream services, queues, and third-party stubs match prod. Pin image SHAs, not latest tags — moving tags hide subtle regressions and break reproducibility for incident replay.

Capture the deployed git SHA so the run is reproducible. Tag the build (e.g., v2024.45.0-rc.1) and confirm the SHA matches the candidate cut from the release branch.

Use a recent prod snapshot with PII scrubbed per your data-handling policy (HIPAA / GDPR / customer DPAs). Stale fixtures are the leading source of false-positive failures from schema drift.

Run a restore drill into a scratch database. A backup that has never been restored is theater; SOC 2 auditors look for evidence of periodic restore tests, and the regression cycle is a good place to capture it.

Cover boundary conditions: empty collections, max-length strings, unicode, leap-second timestamps, multi-tenant isolation. These are the cases real users hit that synthetic happy-path data misses.

Pull the list of merged PRs and epics since the last release. Confirm the test plan has coverage for each user-visible change, infrastructure migration, and feature flag flip planned for this deploy.

Walk the suite with the feature owner. Cases that test deprecated flows should be archived, not left to fail mysteriously next cycle and waste an SDET hour.

Note cases that need rewrite (UI selectors, API contracts) and gaps where new behavior has no coverage. Hand the list to the SDETs before automated-suite maintenance kicks off.

Trigger the full pipeline (Jest / pytest / Playwright / Cypress) against the candidate build. Don't run locally — CI parity matters because flakes correlate with environment differences the local box hides.

Walk the failure list and fix selectors, API contracts, and assertion drift. Distinguish real regressions from script staleness — only the former blocks release; the latter blocks the SDET.

Move flakes to a quarantine suite with a Jira / Linear ticket and a named owner. Untagged flakes get auto-rerun and silently mask real regressions for months.

Use the standard load profile (e.g., 500 RPS for 10 minutes across the top 5 endpoints). Match staging instance sizes to prod or note the discrepancy explicitly in the report.

Pull the dashboard from the previous regression cycle (Datadog / Grafana). A 10%+ degradation at p99 is the threshold most teams treat as a release blocker; tune the threshold to your SLO error budget.

File a ticket with the failing endpoint, the regression delta, and a flame graph or APM trace. Tag the service owner from CODEOWNERS — do not let the ticket float without an assignee until the go/no-go meeting.

Filter the tracker for defects marked Fixed since the last regression. Each one needs a retest against this candidate, not a "trust the developer" sign-off.

Reproduce the original repro steps and confirm the bug no longer occurs. Retest in the same browser, OS, and tenant configuration from the original report — fixes that pass on Chrome can still fail on Safari.

Mark each ticket Verified or Reopened with a comment linking to the test run. Reopened defects route back to the developer and block release until re-fixed.

Walk the top user journeys (login, checkout, primary CRUD flows, billing). Coverage on revenue-impacting paths is non-negotiable; long-tail features can ride lower coverage with documented risk.

Cover each feature behind a flag that's flipping on this release. Include a kill-switch revert test so the rollback path is exercised before you need it at 2am.

Rank gaps by likelihood × impact. Document the accepted risk in the test plan so post-release issues aren't a surprise to the release captain or to engineering leadership.

Run the suite against the release candidate in staging. Capture the CI run ID — downstream defect tickets and the report will reference it for traceability.

Attach the failing run's logs, screenshots, and HAR files to each defect. "It failed once" is not a defect report; reproducible failures with artifacts are.

Each defect gets steps to reproduce, expected vs. actual, environment, and severity. Severity follows the team's rubric — SEV1 blocks release, SEV2 blocks the next release, SEV3 is backlog.

Walk the new defects with the release captain, eng lead, and product owner. For each: block release, fix forward with a known workaround, or accept risk with a tracking ticket.

Look for clusters — same service, same user path, same data shape. Patterns point at root causes (a flaky upstream, a leaked migration, a noisy dependency) that single-defect investigations miss.

Summarize pass/fail counts, performance deltas, defects by severity, accepted risks, and a recommended go/no-go. The report goes to the release captain, eng leadership, and product owner before the sign-off meeting.

Walk the report with the team. Capture the decision, any conditions (must-fix before deploy, monitoring to add, customer comms), and a signed acknowledgement from the release captain.