INC-2025-203: Checkout Service Sev3 Incident

Date: 2026-04-26 Service: Checkout Service Severity: Sev3 Incident Commander: Morgan Lee Jira References: ACME-2005, ACME-2006 GitHub References: GH-2005, GH-2006 API References: API-CHECKOUT-04

Summary

The Checkout Service experienced customer-visible degradation caused by out-of-order event delivery from the shared event bus. The incident affected dependent paths that include Authentication Service, Checkout Service, Billing Service, Inventory Service, and Notification Service, with the strongest impact localized to Checkout Service.

Timeline

• Detection: synthetic checks and service SLO burn alerts fired.
• Triage: on-call engineers reviewed traces, dependency dashboards, and recent deployments.
• Mitigation: traffic was shifted, retries were reduced, and the affected worker pool was drained.
• Recovery: error rate returned to baseline after remediation was deployed and queues caught up.

Root Cause

The proximate cause was out-of-order event delivery from the shared event bus. The deeper contributing factor was incomplete validation of the Checkout Service operational contract against API API-CHECKOUT-04 and insufficient pre-release checks for high-cardinality traffic.

Customer Impact

Customers saw intermittent failures, elevated latency, or delayed asynchronous updates depending on where their request crossed the Checkout Service boundary. No permanent data loss was identified.

Remediation

• Track remediation in Jira tickets ACME-2005, ACME-2006.
• Implement code and test changes in GitHub items GH-2005, GH-2006.
• Update the relevant runbook for Checkout Service and replay this scenario in staging.