SNAT Port Exhaustion (NAT Gateway Outbound Limits).

🎬The scene

3am page. Your payment service — 40 pods, private subnet, one NAT gateway — starts throwing:

Error: connect ETIMEDOUT 34.x.x.x:443  (api.stripe.com)

CPU is 20%. Memory fine. Upstream status page green. Inbound traffic is healthy. Only outbound calls fail, and only to your busiest dependency. Traffic grew from 500 to 2,000 req/s an hour ago. Retries are on, and things are getting worse, not better.

🔍What's actually happening

Every packet leaving a private subnet gets source-NAT'd: the gateway rewrites the source to its own public IP + a source port it picks. To demux replies, each live connection needs a unique 5-tuple:

(src IP, src port, dst IP, dst port, protocol)

Your src IP is the NAT gateway's. The dst is one IP on port 443. Four of five fields are frozen — so the only thing that can vary is the source port.

That gives you ~64,512 ports. Per destination IP:port. Not per instance, not per pod, not per gateway.

📉The math nobody does

It's worse than 64k concurrent, because a closed port isn't instantly reusable. AWS NAT Gateway holds it ~350 seconds before recycling.

64,512 ports / 350 s ≈ 184 NEW connections/sec, sustained, to one destination IP

At 2,000 req/s with a fresh TCP+TLS connection per request, you blow through that in under a minute. Then allocation fails and you get a timeout that looks exactly like "the upstream is down."

🚨The one metric that names it

CloudWatch → NAT Gateway → ErrorPortAllocation. Non-zero is a diagnosis, not a hint. (Azure calls it SNAT port exhaustion; watch SNATConnectionCount / failed flows.)

🔧Fixes, in order of leverage

1️⃣ Reuse connections. One keep-alive socket serves thousands of requests and consumes one port.

• Node: `new https.Agent({ keepAlive: true, maxSockets: 50 })` — and check the lib you use isn't creating its own agent.
• Python: a `requests.Session()`, not `requests.get()` in a loop.
This alone usually ends the incident.

2️⃣ Spread destination IPs. The limit is per dst IP. If the upstream resolves to 8 A records and you actually rotate through them, you get 8× headroom — so don't pin one resolved IP in config.

3️⃣ Add NAT gateways / more EIPs — one per AZ. Also cuts cross-AZ data charges.

4️⃣ VPC endpoints for S3, DynamoDB, ECR, Secrets Manager. That traffic skips NAT entirely, freeing ports and dodging the ~$0.045/GB processing fee.

⚠️Two traps

• The wrong fix: tuning net.ipv4.tcp_fin_timeout or TIME_WAIT reuse on your nodes. Useless here — the exhausted port is held by the NAT gateway, not your kernel.
• The amplifier: retries. A port-allocation timeout triggers a retry, which demands another port. Retrying an exhaustion error is pouring water on an oil fire — you need a circuit breaker or retry budget on this path.

🧪Your exercise

Pick your service's busiest outbound dependency. On one box:

ss -tan | grep ':443' | awk '{print $1}' | sort | uniq -c

Lots of TIME-WAIT and few long-lived ESTABLISHED = you're opening a connection per request. Now estimate new-connections/sec × 350 and see how much of your 64,512 you're really using. Is keep-alive actually on, or just assumed?