Incident: Authentication Outage

Date: 2026-03-18 Service: Authentication Severity: Sev2

Customers experienced failed sign-ins for 22 minutes after a JWKS rotation published the new key before all API nodes had loaded the private key. Checkout and Billing rejected valid tokens because cached key refreshes received inconsistent responses.

Root cause was a deployment ordering bug in the key-rotation job. Remediation included two-phase key activation, canary validation for JWKS endpoints, and an alert on token validation failure rate.