Autoscaling on the Wrong Signal (CPU vs Queue Depth).

🧠The myth

"Our workers autoscale on CPU at 70%. If the queue backs up, Kubernetes adds pods."

⚡Actually…

A worker that spends its life waiting on I/O never reaches 70% CPU. Your queue can climb to 200k messages while the HPA reports 12% CPU and happily keeps 2 replicas.

CPU measures how hard you are burning, not how much work is waiting. For I/O-bound workers those are different numbers.

🔢Concrete

One pod handles 5 msg/s. Each message = 180ms waiting on a payment API + 20ms of actual CPU. At full throughput that pod uses ~10% of a core.

Traffic triples to 15 msg/s:

• CPU: ~30% → HPA does nothing
• Backlog: growing 10 msg/s → 36,000 messages an hour
• Dashboard: green. Users: waiting.

📏The right signal

Scale on backlog per replica, not on burn.

desired = ceil(replicas × backlog_per_replica / target_per_replica)

KEDA version:

triggers:
- type: aws-sqs-queue
  metadata:
    queueURL: https://sqs.../jobs
    queueLength: "20"     # target backlog PER replica

2,000 messages in queue ÷ 20 = 100 pods (clamped by maxReplicaCount).

🎯How to pick that target (don't guess it)

queueLength is really "seconds of work each pod is allowed to hold." A pod draining 5 msg/s with a target of 20 is carrying 4 seconds of backlog. Want burst absorbed in ~10s? target ≈ 50. That is the whole calculation — it is a latency budget, not a magic number.

⚠️Four things that bite people here

• Downstream is the real ceiling. 100 pods hitting a payment API with a 50-connection pool doesn't drain the queue, it relocates it — now as timeouts. Set maxReplicas to what downstream actually tolerates, not to what the math wants.
• Scale-down eats in-flight work. A pod killed mid-message loses it unless you handle SIGTERM, finish the current job, and ack only after success. Set terminationGracePeriodSeconds above your p99 processing time.
• Metric lag is real. HPA polls ~15s; scale-down stabilization defaults to 300s. Expect to react to a burst 30–60s late, so keep headroom rather than tuning for a perfect fit.
• Two metrics, two jobs. Scale on queue depth (the control signal). Alert on oldest-message-age (the SLO signal). Depth alone lies when one poison message is stuck at the head; age catches it.

🧯Bonus: scale-to-zero

KEDA can take you to 0 replicas when the queue is empty — great for cost, bad for latency: the next message pays a full pod cold start. Fine for nightly batch, not for anything a user is waiting on.

🧪Your exercise

Your queue is 30,000 messages deep. Each pod drains 5 msg/s. You want it clear in 5 minutes.

1. How many pods? (do the arithmetic — and then redo it assuming 200 msg/s is still arriving)
2. When you actually run that number, what breaks first: the database connection limit, the third-party API rate limit, or your node pool? Go look up which of those you'd hit and at what replica count.

That second answer is your real maxReplicas.