System Testing Checklist

Verify the staging VPC, IAM roles, and database credentials do not overlap with production. A common gotcha: a staging Lambda still pointing at the prod RDS endpoint because someone copied an env var. Spot-check the secrets manager paths and the outbound network ACLs before kicking off tests.

Confirm OS version, Postgres major version, Redis version, Kubernetes node image, and feature flag defaults match prod. Drift on minor versions (PG 15.4 vs 15.7) is fine; drift on major versions invalidates the test run.

Use the nightly anonymized snapshot, not synthetic fixtures — synthetic data hides cardinality and locale issues. Confirm PII scrubbing ran (no real emails, names hashed) before exposing the dataset to non-engineering testers.

Pull the fix-version filter for the release candidate and walk the linked PRs. Flag any story without acceptance criteria back to the PM before writing test cases — a fuzzy AC is the most common reason a defect gets bounced as 'works as designed' later.

Add or amend Playwright / Cypress cases for new user stories; mark superseded cases as deprecated rather than deleting (audit trail). Tag flaky cases with a quarantine label so they run but don't gate the suite.

Cover empty state, single record, max-page-size, unicode names, RTL locales, and the largest plausible tenant. The unicode + RTL cases are the ones consistently missed and the ones support tickets get filed against.

Trigger the suite from CI against the tagged release candidate (e.g., v2024.45.0-rc.1), not against latest main. Capture the run ID and any quarantined-test results separately so the report is reproducible.

Time-box charters at 60-90 minutes per tester per feature. Take notes in a session log; file any unexpected behavior as a defect even if it doesn't violate a written AC — the AC may be incomplete.

Mark Pass only if every required case passed and any quarantined failures were investigated. Pass-with-issues is the right call when a SEV3 was found that the release captain has agreed to ship around with a follow-up ticket.

Hit 2x peak prod RPS for 15 minutes against the critical user paths. Watch p95 and p99 latency, error rate, and DB connection pool saturation in Datadog. A green load test where p99 doubles is still a fail — eyeball the dashboard, don't trust the exit code.

Trigger Semgrep / CodeQL on the RC sha and review Snyk or Dependabot output for new criticals. Any new CVE rated CVSS 7.0+ blocks the release unless the AppSec lead signs off on a documented exception.

Run axe-core in CI against each changed route; manually keyboard-navigate critical paths. The EU Accessibility Act (effective June 2025) makes this non-optional for products sold into the EU.

Hit Stripe, Auth0/Okta, Segment, and any sandbox endpoints used in the changed code paths. Confirm webhooks deliver and signatures verify; a silent webhook failure won't show in the regression suite.

Publish synthetic events through SQS / Kafka / RabbitMQ and confirm consumers process them with the expected schema. Check the DLQ count before and after — silent schema drift surfaces here first.

Apply the migration to a clone seeded with prod-sized data, time it, and confirm reversibility. ADD COLUMN with a default on a 50M-row table is a classic foot-gun: it takes an exclusive lock and the deploy stalls. Capture wall-clock duration in the data field.

Each ticket needs build sha, browser/OS, exact repro steps, expected vs actual, and a screenshot or HAR. Tickets without a sha get bounced — the first thing engineering will ask is which build it reproduced on.

SEV1 = data loss or auth bypass, blocks release. SEV2 = critical path broken, blocks release unless workaround exists. SEV3 = ships with a follow-up. Triage with the release captain and a PM in the room — severity is a business call, not a QA call.

If any SEV1/SEV2 remains open, the release does not promote — the team cuts a new RC. If all blockers are closed or accepted, proceed to closure.

Tag the next RC (e.g., v2024.45.0-rc.2), notify the release captain in #engineering, and restart functional regression against the new sha. Do not cherry-pick fixes onto the prior RC tag — re-cut from the release branch.

Run the original repro steps verbatim and add a regression case to the automated suite so the same defect cannot ship again. Close the ticket only after the regression case is committed.

Post to Confluence: cases run, pass rate, defects opened by severity, accepted-with-follow-up tickets, performance deltas vs prior release. Link the CI run, the load test dashboard, and the migration timing.

QA lead signs off only after the summary is reviewed by the release captain. This signature is the SOC 2 change-management artifact auditors ask for; do not skip it even on a quiet release.

Walk what slipped past the suite, which charters surfaced the most defects, and where flaky tests masked real failures. File action items in Jira with named owners; an action item without an owner does not get done.