Degraded performance
Webhook deliveries delayed
Webhooks were delivered up to 41 minutes late.
- Status
- ResolvedResolved — last update 17 Nov 2025, 09:20 UTC
- Started
- 17 Nov, 08:39Monday 17 November 2025 UTC
- Duration
- 41 minEnded 17 Nov 2025, 09:20 UTC
- Components
- 1Webhooks and notifications
Timeline
What we posted, as we posted it.
Resolved4 updates · UTC
- ResolvedThis incident has been resolved. Every queued event was delivered in order.17 Nov, 09:20
- MonitoringThe backlog cleared at 09:12 UTC and deliveries are current. We are monitoring.17 Nov, 09:12
- IdentifiedA webhook consumer crashed on a malformed endpoint response and restarted in a loop, leaving a backlog of about 11 thousand events. The consumer has been patched.17 Nov, 08:51
- InvestigatingWebhook deliveries are delayed by several minutes. Events are queued, not lost. Reclaim notices continue to be published on the instance metadata endpoint on time.17 Nov, 08:39
Affected components
- Webhooks and notificationsControl planeOperational
Details
- Type
- Degraded performance
- Started
- 17 Nov 2025, 08:39 UTC
- Ended
- 17 Nov 2025, 09:20 UTC
- Impact
- Webhooks were delivered up to 41 minutes late.
- Reference
- 7e7398c17b59