Build Strands Agents with SageMaker AI models and MLflow

This article demonstrates how to build AI agents using Strands Agents SDK with models deployed on Amazon SageMaker AI endpoints, integrated with MLflow for observability and A/B testing.

• Deploy foundation models from SageMaker JumpStart as SageMaker AI endpoints
• Integrate Strands Agents SDK with SageMaker AI models using OpenAI-compatible APIs
• Set up SageMaker Serverless MLflow App for automatic agent tracing and observability
• Implement A/B testing by deploying multiple model variants behind single endpoint
• Evaluate agent performance using MLflow GenAI evaluation framework with custom scorers
• Compare model variants (Qwen3-4B vs Qwen3-8B) using metrics and LLM-based judges
• Transition to better-performing model by adjusting traffic weights

This approach enables organizations to build production-grade AI agents with infrastructure control, cost optimization, compliance capabilities, and comprehensive monitoring without relying solely on managed foundation model services.