Home icon

How Airbnb uses cost per task to operate coding agents on Amazon Bedrock

Machine Learning Blog



This article explains how Airbnb evaluates and operates coding agents on Amazon Bedrock, using cost per task as a key metric to guide model selection and monitor production performance.

  • Run evals on representative tasks to measure model score and cost per task before rollout
  • Use CloudWatch metrics (latency, token counts, throttling) to diagnose production issues and compare with eval results
  • Monitor four token types: uncached input, cache reads, cache writes, and output, each priced differently
  • Identify equivalent models across families to rebalance traffic when capacity becomes constrained
  • Attribute spend by team, user, application, and feature using IAM roles, tags, and request metadata
  • Adjust reasoning effort settings and max_tokens to balance quality, cost, and quota consumption
  • Compare production metrics (cost per request, requests per user) against eval predictions to validate model choices

By combining structured evals with production observability, Airbnb can make rapid deployment decisions and optimize costs when model behavior differs from expectations.



Go to article

The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.

Related articles

Nov 12
2026
Optimizing cost per task and proving agent ROI on Amazon Bedrock
Nov 13
2026
Cost attribution and controls for AI developer and productivity tools on Amazon Bedrock
Nov 17
2026
Reducing token costs and latency in Amazon Bedrock AgentCore JSON workloads
Nov 17
2026
Building a gated review pipeline for AI coding agents on AWS

The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.