Spot Instances — Interruptions Are About Capacity, Not Price
Interruptions Are About Capacity, Not Price
Cloud infrastructurecloud infrastructure / cost & resilience
The myth
"Spot is an auction. You set a max price, and if the market price rises above your bid you get outbid and killed. So bid max = on-demand price and you're basically safe."
Actually
Since 2017 AWS spot prices are smoothed — they drift slowly with long-term supply and demand, and everyone in a pool pays the same current price. Bidding high buys you almost nothing. You get reclaimed when EC2 needs that capacity back for on-demand/reserved customers. Interruption is a capacity event, not a pricing event.
The unit that actually matters: the pool
A capacity pool = instance type × size × AZ × OS.
m6i.large/us-east-1a and m6i.large/us-east-1b are different pools. So are m6i.large and m6i.xlarge in the same AZ.
Interruption risk is per-pool. Running one instance type in one AZ is the #1 reason teams say "we tried spot, it didn't work."
The warning you get
- AWS: 2-minute termination notice (plus an earlier "rebalance recommendation")
- GCP preemptible / Azure spot: ~30 seconds
Poll it from the instance:
curl -s -H "X-aws-ec2-metadata-token: $T" \
http://169.254.169.254/latest/meta-data/spot/instance-action404 = healthy. 200 with {"action":"terminate","time":"..."} = you have under 120s. Or catch the EventBridge event cluster-wide instead of polling per node.
Three rules that make spot production-grade
1️⃣ Diversify. Use capacity-optimized allocation across 8–10+ instance types and every AZ. Breadth of pools beats picking "the cheap one."
2️⃣ Make 120 seconds actually enough. Deregister from the load balancer first, and check your deregistration delay — the default is 300s, so a 2-minute handler silently fails to drain. Checkpoint long jobs. Ack queue messages after work completes, never before, so in-flight work is redelivered rather than lost.
3️⃣ Mix. On-demand base for your floor capacity, spot for the burst on top.
Numbers worth knowing
70–90% cheaper. Well-diversified pools often sit under ~5% interruptions/month. AWS publishes per-pool interruption frequency and a Spot placement score — check them before choosing types, not after.
The gotcha nobody plans for
"Lost an instance" is the easy failure. The hard one is "can't get one." Spot capacity for a new launch can simply be unavailable for hours. If your autoscaler can only scale out on spot, a capacity crunch looks exactly like an outage. Always have a fallback to on-demand.
Your exercise
Take your heaviest stateless workload and write down two numbers:
(a) seconds from SIGTERM to safely gone — including LB deregistration
(b) how many distinct instance-type × AZ pools it can run on today
If (a) > 120 or (b) < 4, you just found your two fixes. Which one is worse in your setup?