Security Incident & Breach Response, Key Compromise

Panic causes two classic mistakes

When a key or credential is compromised, panic causes two classic mistakes: wiping systems immediately (destroying the evidence you need) and doing a hard key cutover (logging every user out). A senior responder runs a disciplined loop instead.

The NIST-style loop

Detection  -> know something is wrong
Containment-> stop the bleeding without destroying evidence
Eradication-> remove the foothold, rotate secrets
Recovery   -> restore trusted state, watch for reinfection
Lessons    -> blameless postmortem, fix root cause

Detection. Feed everything into centralized logging or a SIEM (Splunk, Elastic, a cloud-native equivalent) and alert on anomalous key usage: geo-velocity impossibilities (the key signs from two continents a minute apart), unusual volume, or calls at odd hours. Seed honeytokens (a fake credential that should never be used, so any use is a certain intrusion signal). And plan for the humbling reality that an outside party (a researcher, a customer, law enforcement) often notifies you first, so build an intake path for external reports.

Containment without destroying evidence. Isolate affected systems (pull them from the load balancer, cut network egress) and revoke active sessions, but do not wipe yet. This is the phase where the discipline matters most.

Rotate a widely-used key without downtime

If one signing key protects every session and you just delete it, every valid token instantly becomes invalid and the whole userbase is logged out. The answer is to design for overlapping key validity ahead of time. Publish keys via a JWKS (JSON Web Key Set) endpoint with a key id (kid) in each token header. To rotate: (1) add the new key to the JWKS so verifiers accept both old and new, (2) flip signing to the new key, (3) after tokens signed with the old key have expired, remove the old key. Because verifiers trust both during the overlap, nobody is logged out. Under compromise you compress this: shrink token TTLs immediately so old tokens age out fast, force re-authentication for genuinely affected sessions, and pull the compromised kid from the JWKS. Short-lived credentials from a secrets manager (Vault, cloud KMS) make this routine rather than heroic.

Eradication and recovery. Remove the attacker's foothold, rotate all potentially-exposed secrets, then restore from a known-good state and watch closely for reinfection. This is where immutable, object-locked backups pay off: if the attacker also ran ransomware, a ransomware-resistant backup is your clean recovery path.

Forensics and the legal clock, in parallel

The instant you suspect a breach, evidence preservation starts: snapshot affected volumes and preserve immutable logs before you clean anything, and maintain chain of custody so the evidence holds up later. Simultaneously the regulatory clock starts: GDPR requires notifying the supervisory authority within 72 hours of becoming aware of a qualifying breach, and affected users without undue delay if there is high risk. So legal and comms are named roles in the runbook, activated at hour zero, not consulted after cleanup.

Interview nuance: the trap is optimizing for "fix it fast." Wiping and rebuilding immediately feels decisive but destroys forensics and, with a hard key cutover, causes a self-inflicted outage on top of the breach. The strong answer sequences containment before eradication, rotates keys via overlapping validity, and runs forensics and the 72-hour legal clock in parallel from the start.

Recap: run detection, containment, eradication, recovery, lessons in order; rotate the compromised key via overlapping JWKS validity plus shortened TTLs so nobody is logged out; preserve evidence before cleanup with chain of custody; and start the GDPR 72-hour notification clock the moment you become aware.