Handling a reclaim, in code.

The platform never deletes a reclaimed instance: it stops it, keeps the persistent disk attached and keeps the reserved address. Your job is to make sure the last 2 minutes are spent writing something you can resume from.

What is guaranteed#

• 2 minutes of notice before a planned reclaim, on three channels at the same moment.
• The instance is stopped, not deleted. Disk, data, IP and configuration survive.
• Compute billing ends at the stop. The notice window itself is billed as running time; nothing after it is.
• No notice is sent for a hardware failure — that is an incident, not a reclaim, and it is handled under the SLA for on-demand instances.

The metadata endpoint#

Every instance can reach a link-local metadata service that is not routable from outside. Poll it every 5 to 15 seconds; it costs nothing and it is the only channel that works even if your webhook endpoint is down.

$ curl -s http://169.254.169.254/v1/notice
{"reclaim_in": null, "state": "running", "instance": "i-7f3a2c"}

{
  "reclaim_in": 120,
  "reclaim_at": "2026-09-15T18:14:40Z",
  "state": "reclaim_notice",
  "instance": "i-7f3a2c",
  "reason": "capacity",
  "relaunch": true
}

{
  "instance": "i-7f3a2c",
  "gpu": "h100-sxm",
  "gpu_count": 1,
  "tier": "spot",
  "region": "us-east",
  "template": "pytorch",
  "volume": {"ref": "vol-4b19c7", "mount": "/mnt/vol", "size_gb": 500},
  "started_at": "2026-09-15T17:58:04Z"
}

The webhook#

Register an HTTPS endpoint in the console and lifecycle events are POSTed to it as JSON. Use it when the decision belongs to your scheduler rather than to the job — for example to start the same work somewhere else while the instance finishes its checkpoint.

{
  "event": "instance.reclaim_notice",
  "sent_at": "2026-09-15T18:12:40Z",
  "data": {
    "instance": "i-7f3a2c",
    "gpu": "h100-sxm",
    "region": "us-east",
    "reclaim_in": 120,
    "reclaim_at": "2026-09-15T18:14:40Z",
    "relaunch": true
  }
}

Events: instance.running, instance.reclaim_notice, instance.stopped, instance.relaunched, instance.terminated, balance.low. Each delivery carries a signature header computed from your webhook secret; verify it before acting, and answer 2xx within five seconds — deliveries are retried with a backoff for up to an hour.

The checkpoint pattern#

The pattern has three parts and does not depend on our platform: write state you can resume from, notice the warning, exit cleanly.

import threading, time, requests

NOTICE = "http://169.254.169.254/v1/notice"
_flag = threading.Event()

def _watch(interval=10):
    while not _flag.is_set():
        try:
            r = requests.get(NOTICE, timeout=2).json()
            if r.get("reclaim_in") is not None:
                _flag.set()
        except Exception:
            pass          # a failed poll is never a reason to stop training
        time.sleep(interval)

def start():
    threading.Thread(target=_watch, daemon=True).start()

def reclaiming():
    return _flag.is_set()

import spot_guard, torch, os

CKPT = "/mnt/vol/run-42/latest.pt"   # on the persistent volume, never on scratch
spot_guard.start()

start_step = 0
if os.path.exists(CKPT):            # resume, do not restart
    state = torch.load(CKPT)
    model.load_state_dict(state["model"]); opt.load_state_dict(state["optim"])
    start_step = state["step"]

for step in range(start_step, total_steps):
    train_one_step()
    if step % 200 == 0 or spot_guard.reclaiming():
        torch.save({"model": model.state_dict(),
                    "optim": opt.state_dict(),
                    "step": step}, CKPT + ".tmp")
        os.replace(CKPT + ".tmp", CKPT)     # atomic: never a half-written file
    if spot_guard.reclaiming():
        break                                 # exit before the machine is stopped

With common trainers#

Most training stacks already checkpoint on a schedule and resume from the newest one. Point their output at the volume and the job is spot-ready with no extra code:

Interruptible serving#

Serving on spot works when a load balancer in front can drain a node. On the notice: stop accepting new requests, let in-flight ones finish, deregister, exit. With two spot replicas in different regions and an on-demand baseline, a reclaim becomes a rolling restart instead of an outage.

Automatic relaunch#

Turn on auto-relaunch when configuring an instance and the platform queues it for the next free GPU of that model, with the same volume, the same template and the same entrypoint. You keep paying nothing while it waits. The instance reference changes; the volume reference does not, which is why checkpoint paths should not contain the instance reference.

If you would rather decide yourself, leave it off and handle instance.stopped in your scheduler: relaunch on the same model, on a different model, or on the on-demand tier.

Testing your handler#

Do not wait for a real reclaim to find out that your checkpoint path was wrong. Two ways to test:

1. Stop the instance from the console while the job runs, then relaunch it. A manual stop takes the same code path as a reclaim: the disk stays, the process is signalled, the resume must work.
2. Fake the notice locally by pointing NOTICE at a file or a local server that returns {"reclaim_in": 120}. Your handler should write a checkpoint and exit within a few seconds.

Three expensive mistakes#