AWS's Application Load Balancer Target Optimizer, announced in November 2025, runs an agent on target instances to manage concurrency. A developer tested if this enables auto-scaling for LLM applications, where instances handle only one or two requests. The setup used instances with TARGET_CONTROL_MAX_CONCURRENCY set to 1 for controlled load testing.

Scaling proved reactive, triggered by CloudWatch metrics like TargetControlRequestRejectCount. This metric spikes when the ALB rejects requests due to full capacity, which also means users already see 503 errors. Linking this alarm to an Auto Scaling Group confirmed scale-out, but the approach impacts user experience before scaling occurs.

For LLM workloads with heavy, uneven requests, this agent-based control offers finer granularity than traditional ALB methods. However, the reactive nature requires client-side retries or better predictive metrics. Future iterations may offer proactive scaling, a notable evolution for ALB traffic management.