Incident Response Checklist

The on-call engineer creates the incident in PagerDuty (or Opsgenie / FireHydrant), assigns an incident commander, and opens a dedicated Slack channel using the #inc-YYYYMMDD-name convention. Do not investigate in the alerting channel — noise drowns the timeline.

Use the severity matrix: P1 = customer-impacting outage or confirmed data exposure; P2 = major degradation or suspected breach; P3 = limited impact with workaround; P4 = no customer impact. Severity drives paging, comms cadence, and exec notification — err high; you can downgrade later.

Cross-check the originating signal against a second source — EDR alert against SIEM logs, monitoring against synthetic checks, user report against access logs. Tuned-out detection rules and stale dashboards are common false-positive sources.

The IC drives decisions and is not hands-on-keyboard. The scribe maintains the running timeline in Slack with timestamps. The comms lead handles status-page updates and stakeholder notifications. On a P1, all three roles must be filled by separate people.

Snapshot affected EC2 / Azure VMs before terminating. Capture memory dumps from EDR (CrowdStrike RTR, SentinelOne) where supported. Export relevant SIEM queries with time ranges locked. Once you reboot or wipe, volatile evidence is gone — and so is the chain of custody for any later legal action.

Query Splunk / Datadog / Sumo for authentication events, network flows, and process executions across the suspected window. Pull EDR detections, IdP sign-in logs (Okta / Entra ID), and CloudTrail / Azure Activity Log entries. Default cloud retention often falls below SOC 2 / PCI minimums — pull now, archive to cold storage.

Enumerate every host, identity, SaaS tenant, and data store touched by the incident. For credential compromise, check sign-in logs across every IdP-connected app for the affected accounts. Scope drives breach-notification thresholds — undercounting here causes regulatory exposure later.

Use CrowdStrike network containment, SentinelOne disconnect, or Defender for Endpoint isolation rather than pulling cables — the agent stays online for forensics while blocking lateral movement. For cloud workloads, modify the security group to deny all egress except to your forensic jump host.

In the IdP, force sign-out and reset for every implicated account. Rotate API tokens, OAuth grants, and SSH keys associated with the identity. For service accounts, rotate the secret in Vault / Secrets Manager and redeploy. SMS / email MFA is bypassable via SIM swap and phishing — escalate affected users to FIDO2 / passkey before re-enabling.

For confirmed compromise, rebuild from a known-good image rather than cleaning in place — scheduled tasks, registry run keys, cron jobs, and rogue IAM roles are easy to miss. Validate the gold image predates the initial compromise based on the timeline.

Identify the CVE or misconfiguration that enabled initial access. Push the patch through your patch management tool (Action1, Automox, Intune) to all hosts running the affected version, not just the compromised one. Re-scan with Tenable / Qualys / Wiz to confirm closure.

Restore from immutable backups (Veeam hardened repos, AWS Backup vault lock, Rubrik). Verify the restore point predates compromise based on your timeline — restoring from a backup taken after initial access reintroduces the foothold. Validate RPO and RTO against SLA commitments.

For confirmed security incidents, loop in legal before drafting external communications — attorney-client privilege over the investigation depends on counsel directing the work. Privacy counsel evaluates breach-notification triggers under GDPR Article 33, HIPAA, and applicable state laws.

GDPR requires notification to the supervisory authority within 72 hours of awareness. State breach-notification statutes range from 30-90 days, with varying recipient lists (AG, credit bureaus, affected residents). HIPAA breach of >500 individuals requires HHS notification within 60 days. Track each filing separately with submission timestamps.

Hold the postmortem within 5 business days while the timeline is fresh. Invite the IR team, the service owner, legal (for security incidents), and an exec sponsor. Blameless framing focuses on systems and processes, not individuals — the goal is durable fixes, not blame.

Capture the technical root cause, the contributing factors, MTTD (time from first malicious action to detection), and MTTR (detection to full recovery). Trend these across incidents quarterly to know whether your detective controls are improving.

Every action item lands in Jira / Linear with a named owner and a due date. Track completion in monthly security ops review — postmortem actions that don't ship are the most reliable predictor of the same incident recurring.