Right-size generative AI endpoints with concurrency sweeps on Amazon SageMaker AI

This article explains how to use concurrency sweeps on Amazon SageMaker AI to right-size generative AI endpoints by finding the optimal balance between throughput and latency.

• Concurrency sweeps send controlled, increasing levels of concurrent traffic to measure throughput and latency at each level
• Deploy models using native vLLM containers with configurable environment variables for GPU memory and caching
• Define workload profiles specifying input/output token counts and streaming mode to simulate realistic traffic patterns
• Use CreateAIBenchmarkJob API to run sweeps and identify the saturation point where throughput plateaus and latency spikes
• Automated max-concurrency-under-sla recipe searches for highest concurrency meeting multiple SLA thresholds
• Analyze results by plotting throughput vs. p99 latency to find the "knee" indicating optimal operating region

Concurrency sweeps replace guesswork with data-driven insights for capacity planning, enabling confident fleet sizing before production deployment.