Your EKS Cluster Will Run Out of IPs Before It Runs Out of Compute (Part 1)

The Day the Cluster Had Room but No Addresses

It started the way these things usually do: workloads scaling up during business hours, and suddenly new pods weren't coming up. Not crashing - just not starting. The nodes had free CPU. Free memory. The autoscaler wasn't complaining about capacity.

The events told a different story:

failed to assign an IP address to container
add cmd: failed to assign an IP address to pod

# and on the AWS side, node scale-ups failing with:
InsufficientFreeAddressesInSubnet

The cluster wasn't out of compute. It was out of IP addresses. And once you understand how EKS networking works, the surprising part isn't that it happened - it's that it didn't happen sooner.

Why EKS Eats IP Addresses Faster Than You Think

On most Kubernetes platforms, pods live on an overlay network - their IPs are virtual and effectively unlimited. EKS's default networking (the AWS VPC CNI) works differently, and it's both its best feature and its trap: every pod gets a real, routable IP address from your VPC subnets.

That's great for performance and for integrating with security groups, load balancers, and VPC flow logs. But it means your pods and your nodes are drinking from the same, finite pool. Three things drain that pool faster than intuition suggests:

• Every pod is a real IP. 300 pods = 300 VPC IPs, plus one per node, plus load balancers, plus anything else living in those subnets.
• Nodes hoard IPs in advance. The CNI attaches extra ENIs and keeps a warm pool of pre-allocated IPs on every node so pods can start instantly. Depending on the instance type, a single node can reserve dozens of addresses while running only a handful of pods. From the subnet's point of view, those IPs are gone.
• Small subnets compound the problem. A /24 subnet has 251 usable addresses (AWS reserves five per subnet). Two or three busy nodes' worth of warm pools plus real pods, and a /24 is done.

The Fix We Shipped: Move the Nodes to Bigger Subnets

The proper long-term fix (more on it below) requires touching every node in the cluster, and we were watching subnets sit at 90%+ utilization in production. We needed headroom now. So we did the pragmatic thing: created new, much larger subnets (/20s - about 4,091 usable IPs each) in the same VPC and migrated the node groups onto them.

1Carve the new subnets

We added /20 subnets in each availability zone from unused VPC address space, tagged them for EKS the same way as the originals (the cluster tag and the internal/public ELB role tags), and wired them to the same route tables. Nothing about the cluster changes yet - you're just laying new road.

2Create a new node group on the new subnets

Rather than editing the existing node group in place, we created a fresh node group pointed at the /20s. Same instance types, same labels and taints, same IAM role. This gives you a clean rollback path: the old node group is still there, untouched, until you're happy.

3Cordon, then drain - node by node

We ran a single node group, so there was no elaborate AZ-by-AZ choreography: cordon all the old nodes so nothing new lands on them, then drain them one at a time and let the scheduler rebuild the evicted pods on the new nodes - where they pull their IPs from the roomy /20s.

# stop new pods landing on the old nodes
kubectl cordon -l eks.amazonaws.com/nodegroup=old-nodegroup

# then drain node by node, respecting disruption budgets
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data

Stateless workloads moved without drama. The stateful ones are where you slow down - and one of ours stopped the drain completely.

4The drain that stopped dead: a singleton with a PDB

One of our workloads is Neo4j - a single-replica StatefulSet, because the edition we run isn't horizontally scalable. It was protected by a PodDisruptionBudget with minAvailable: 1. Read that combination slowly: the PDB demands at least one pod stay available, and there is exactly one pod. No eviction can ever satisfy it. The drain hit the pod and refused to move:

evicting pod neo4j/neo4j-0
error when evicting pods/"neo4j-0":
Cannot evict pod as it would violate the pod's disruption budget.

There is no clever zero-downtime answer here. A singleton has to go down to move - the only question is whether that happens in a controlled way or a panicked one. The standard options, in order of preference:

• Temporarily relax the PDB - patch or delete it, drain, then restore it. The most explicit and auditable route: anyone reading the change history sees exactly what was done and why.
• kubectl drain --disable-eviction - drains using direct deletes instead of the eviction API. PDBs only guard evictions, not deletes, so this walks straight past the budget. Same outcome, one flag.
• Delete the pod directly (kubectl delete pod neo4j-0) - the StatefulSet controller rebuilds it on a schedulable node.

We bypassed the PDB in a controlled window and let the StatefulSet bring Neo4j back up on a new node. One caveat if you're doing this with any EBS-backed StatefulSet: the volume is AZ-bound, so the replacement node must be in the same availability zone - otherwise the pod comes back Pending for an entirely different reason.

5Tune the warm pool so nodes hoard less

While we were in there, we also reined in how aggressively nodes pre-allocate IPs:

kubectl set env daemonset aws-node -n kube-system \
  WARM_IP_TARGET=5 \
  MINIMUM_IP_TARGET=10

Instead of grabbing a whole ENI's worth of addresses at a time (WARM_ENI_TARGET=1, the default), each node now keeps a small buffer of warm IPs. The tradeoff is honest: slightly slower pod starts during a big scale-up burst, in exchange for dramatically less IP hoarding. For most workloads you will never notice.

Being Honest: This Is a Bridge, Not the Destination

I want to be upfront about what we did and didn't do. The subnet migration bought us years of headroom, but it didn't change the underlying model: pods, nodes, load balancers, and everything else in the VPC are still competing for the same address space. If the cluster keeps growing - or the VPC gets crowded by other teams - the same wall exists, just further away.

The structural fix on EKS is VPC CNI custom networking, and it's worth understanding even before you need it:

• You attach a secondary CIDR to the VPC - typically from the CGNAT range 100.64.0.0/10, which doesn't collide with your routable address plan.
• You create an ENIConfig per availability zone pointing at new (huge) subnets carved from that secondary range.
• You set AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true on the CNI. From then on, nodes keep their IPs from the primary subnets, but pods get theirs from the secondary CIDR. Pod IPs stop competing with the rest of your VPC entirely.

So why haven't we flipped it on yet? Three reasons, and none of them are secrets:

• Every node must be recycled. Custom networking only applies to nodes launched after it's enabled - it's a full, rolling replacement of the cluster's compute, which we want to schedule deliberately, not bolt onto an incident response.
• It costs pod density. With custom networking, the node's primary ENI no longer hosts pods, so max pods per node drops unless you also enable prefix delegation. That interacts with instance sizing and needs real capacity planning.
• More moving parts. Per-AZ ENIConfigs, a second CIDR in the routing story, and one more thing on the diagram when you're debugging connectivity at 2 AM. Worth it at scale - but you should walk in with eyes open.

Frequently Asked Questions

Why do pods fail to schedule when nodes have free CPU and memory?

Because on EKS's default VPC CNI, every pod needs a real IP from your VPC subnets. When the subnets are exhausted, the CNI can't assign addresses, and pods fail to start regardless of how much compute is free. Check subnet available-IP counts before blaming the scheduler.

How do I see how close my subnets are to exhaustion?

The VPC console shows "Available IPv4 addresses" per subnet, and aws ec2 describe-subnets exposes the same field for alerting. If you run more than a couple of nodes per /24, put a monitor on it - this failure mode gives you very little warning.

Is moving to bigger subnets enough, or do I need custom networking?

Bigger subnets are usually enough for a long time, and they're by far the cheaper change. Custom networking is the answer when pod IPs must stop competing with your routable VPC space entirely - very large clusters, crowded VPCs, or strict address planning.

My drain is blocked by "cannot evict pod as it would violate the pod's disruption budget" - what do I do?

Compare the PDB against the replica count. If the workload has one replica and the PDB demands minAvailable: 1, no eviction can ever succeed - you have to temporarily relax the PDB, use kubectl drain --disable-eviction, or delete the pod directly, accepting the brief downtime. The durable fix is aligning the PDB with reality: more replicas, maxUnavailable: 1, or no PDB on singletons.

Does prefix delegation solve this instead?

Prefix delegation (assigning /28 blocks per ENI) increases pod density per node and reduces per-pod ENI API calls, but the addresses still come out of the same subnets. It changes how fast you drink - not the size of the glass. It pairs well with custom networking rather than replacing it.

My Takeaway

EKS's networking model hands every pod a first-class VPC address - which is wonderful, right up until you discover your address plan was designed for a cluster a quarter of the size. The lesson we took: treat subnet capacity as a cluster resource, exactly like CPU and memory. Monitor it, alert on it, and plan for it - because the failure arrives silently and looks like a scheduler problem until you know where to look.

Migrating to /20 subnets was the right call under pressure: cheap, low-risk, reversible, and it bought us the time to do custom networking properly instead of in a panic. That's Part 2 - and when it happens, you'll read exactly how it went here.

For the official details, see the AWS docs on VPC CNI custom networking and the amazon-vpc-cni-k8s project's configuration reference.