How A/B Testing and Multi-Model Hosting Accelerate Generative AI Feature Development in Amazon Q

This article discusses how the Amazon Q Developer service team leverages A/B testing and multi-model hosting to accelerate the development and deployment of generative AI features.

Specifically, the article covers:

• What is A/B testing and how it helps evaluate new model variants on a subset of users
• How users are consistently assigned to control or treatment groups using hashing
• Techniques for segmenting users for A/B tests based on criteria like IDE used
• Routing traffic to different models hosted on ECS using Application Load Balancer
• How the IDE plugin polls the backend to get the user's group and serve the appropriate experience
• Ingesting telemetry data from the plugin into OpenSearch Serverless for analysis
• The benefits of A/B testing and multi-model hosting in accelerating experimentation and innovation