Partial outage
Metadata endpoint returning errors on some nodes
The metadata endpoint was unavailable on 5 nodes for 35 minutes.
- Status
- ResolvedResolved — last update 22 Sep 2024, 06:23 UTC
- Started
- 22 Sep, 05:48Sunday 22 September 2024 UTC
- Duration
- 35 minEnded 22 Sep 2024, 06:23 UTC
- Components
- 1Instance metadata service
Timeline
What we posted, as we posted it.
Resolved4 updates · UTC
- ResolvedThis incident has been resolved. No spot instance was reclaimed without its notice.22 Sep, 06:23
- MonitoringThe endpoint has been answering on every node since 06:14 UTC. Reclaims on the affected nodes resume with the full notice window. We are monitoring.22 Sep, 06:14
- IdentifiedThe metadata agent on 5 nodes was upgraded to a build that binds to the wrong interface. The previous build is being restored.22 Sep, 06:01
- InvestigatingThe instance metadata endpoint is returning 503 on a subset of nodes. Reclaim notices for those nodes are being delivered by webhook and shown in the console, and reclaims on them are paused until the endpoint is restored.22 Sep, 05:48
Affected components
- Instance metadata serviceControl planeOperational
Details
- Type
- Partial outage
- Started
- 22 Sep 2024, 05:48 UTC
- Ended
- 22 Sep 2024, 06:23 UTC
- Impact
- The metadata endpoint was unavailable on 5 nodes for 35 minutes.
- Reference
- 868491abd058