Cross-AZ Traffic — The Bill Line Nobody Can Explain.

🐛The symptom

Your cluster is healthy. p99 is fine. Then finance forwards the invoice:

EC2-Other / DataTransfer-Regional-Bytes .... $4,180

Regional. Not internet egress. Not NAT. Traffic that never left us-east-1 — and it's your third-biggest line item.

🔍 Step 1: who's talking?

You have 3 AZs (a/b/c), 9 API pods, 3 Redis nodes, 1 primary Postgres in AZ-a.

Every API pod talks to Redis and Postgres on every request. Kubernetes Services load-balance across ALL endpoints with zero awareness of topology. So statistically:

• ~2/3 of every Redis call crosses an AZ boundary
• ~2/3 of every Postgres call crosses an AZ boundary

🔍Step 2: the arithmetic

AWS charges cross-AZ transfer $0.01/GB out AND $0.01/GB in = $0.02/GB round trip.

5,000 req/s × 40 KB avg payload = 200 MB/s
× 2/3 crossing a boundary      ≈ 133 MB/s
× 86,400 s                     ≈ 11.5 TB/day
× $0.02/GB                     ≈ $230/day  →  ~$7k/month

You never wrote a line of code that "sends data to another AZ." The topology did it for you.

⏱️Step 3: it's not only money

Same-AZ RTT: ~0.1–0.3 ms. Cross-AZ RTT: ~0.5–2 ms.

Sounds trivial — until one request makes 30 backend calls. That's ~45 ms of pure geography added to p99, invisible in every flame graph because the CPU is idle the whole time.

🛠️The fix: make routing topology-aware

Kubernetes ships this. On the Service:

spec:
  trafficDistribution: PreferClose

(Or the older internalTrafficPolicy / topology hints, depending on your version.) kube-proxy then prefers endpoints in the caller's own zone and only spills over when local ones are unhealthy.

Non-k8s equivalents:

• Envoy/Istio: zone-aware load balancing (on by default in Istio)
• Postgres read replicas: one per AZ, route reads to the local one
• Kafka: client.rack + RackAwareReplicaSelector → consumers fetch from a same-AZ replica instead of always the leader
• Redis Cluster: READONLY to read from a same-AZ replica

⚠️Where people get it wrong

1. "I'll just run everything in one AZ." That's not cost optimization, that's deleting your availability story. One AZ outage = full outage.
2. Skew. PreferClose sends AZ-local traffic to AZ-local pods. If AZ-a has 5 pods and AZ-c has 1, that one pod gets hammered. Pair it with topologySpreadConstraints so replicas spread evenly.
3. Writes can't be localized. A single Postgres primary lives in one AZ, full stop. Only reads are localizable — and only if you accept replica lag.
4. Cross-AZ ≠ cross-region. Cross-region is ~$0.02/GB and 20–70 ms. Different problem, much bigger numbers.
5. Same-AZ is free ONLY over private IPs. Route through a public IP, an ALB, or a NAT Gateway and you pay again ($0.045/GB NAT processing) even for same-AZ traffic.

🧪Find yours

Cost Explorer → filter UsageType = DataTransfer-Regional-Bytes. In VPC Flow Logs, compare the AZ of srcaddr vs dstaddr.

Rule of thumb: if regional transfer is >5% of your compute spend, you have a topology problem, not a traffic problem.

🎯Your exercise

Take your busiest internal service. Estimate: requests/sec × average response size × (1 − 1/number_of_AZs) × $0.02/GB × 2.6M seconds/month.

Then answer the harder question: which of those calls could safely be served from a same-AZ replica, and which one absolutely must hit the primary? The split between those two is your real architecture.