Stop losing money on Spot interruptions in EKS

AWS says Spot Instances can cost up to 90% less than On-Demand (source). The catch is that AWS can take them back with a two-minute warning. Teams that get burned once often retreat to On-Demand for good and pay full price for workloads that would run fine on Spot.

The fix is to treat interruptions as normal events. Here's the setup.

What goes wrong without handling

• Pods are killed mid-request, causing user-facing errors
• Deploys fail halfway through a rollout
• In-flight jobs are lost
• Several nodes from the same capacity pool get reclaimed at once

1. Diversify instance types

Interruptions happen per capacity pool (instance type in an Availability Zone). The more pools you allow, the less likely you are to lose many nodes at once. Allow whole instance categories and generations rather than naming three instance types.

2. Handle the interruption notice

Pick one mechanism, not two:

• With Karpenter, turn on its native interruption handling. It watches an SQS queue for Spot interruption warnings, rebalance recommendations and instance state changes, then cordons and drains the node and launches a replacement. You point it at the queue with the settings.interruptionQueue Helm value. Karpenter's docs advise against running AWS Node Termination Handler alongside it.
• With managed node groups, EKS handles Spot capacity rebalancing for you.
• With self-managed node groups, run AWS Node Termination Handler in queue mode.

3. PodDisruptionBudgets

A drain respects PDBs, so give every multi-replica service one:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: api

A PDB on a single-replica deployment just blocks the drain. Run at least two replicas for anything on Spot that serves traffic.

4. Shut down cleanly

• Set terminationGracePeriodSeconds shorter than the two-minute notice, leaving room for the drain.
• Handle SIGTERM: stop accepting new work, finish or checkpoint in-flight work, exit.
• Add a short preStop sleep so load balancers stop routing before the process exits.

5. Test it on purpose

AWS Fault Injection Service can send real Spot interruption notices to instances you choose. Run it in staging, watch the dashboards, and only then move production workloads.

Checklist

• Instance diversity across categories and generations
• One interruption handler (Karpenter native, managed node groups, or NTH), not two
• PDBs on every multi-replica service
• Graceful shutdown under two minutes
• A tested interruption in staging

If you want this set up and tested on your cluster, it's part of the EKS Cost Sprint, with every change shipped as a reviewable PR.