Degraded performance
Webhook deliveries delayed
Webhooks were delivered up to 34 minutes late.
- Status
- ResolvedResolved — last update 3 Sep 2025, 13:42 UTC
- Started
- 3 Sep, 13:08Wednesday 3 September 2025 UTC
- Duration
- 34 minEnded 3 Sep 2025, 13:42 UTC
- Components
- 1Webhooks and notifications
Timeline
What we posted, as we posted it.
Resolved4 updates · UTC
- ResolvedThis incident has been resolved. Every queued event was delivered in order.3 Sep, 13:42
- MonitoringThe backlog cleared at 13:36 UTC and deliveries are current. We are monitoring.3 Sep, 13:36
- IdentifiedA webhook consumer crashed on a malformed endpoint response and restarted in a loop, leaving a backlog of about 14 thousand events. The consumer has been patched.3 Sep, 13:15
- InvestigatingWebhook deliveries are delayed by several minutes. Events are queued, not lost. Reclaim notices continue to be published on the instance metadata endpoint on time.3 Sep, 13:08
Affected components
- Webhooks and notificationsControl planeOperational
Details
- Type
- Degraded performance
- Started
- 3 Sep 2025, 13:08 UTC
- Ended
- 3 Sep 2025, 13:42 UTC
- Impact
- Webhooks were delivered up to 34 minutes late.
- Reference
- 10dc5f093e4c