Incident: Billing Failures

Date: 2026-05-14 Service: Billing Severity: Sev1

Billing capture jobs failed for 37 minutes after a payment provider API version changed a nested risk_result field from nullable object to omitted field. Checkout authorizations succeeded, but captures were delayed and invoices remained pending.

The root cause was insufficient contract testing against provider webhooks. Remediation added schema-tolerant parsing, replay tooling, and Promptfoo-style regression cases for billing incident explanations.