Fix evaluation bias to unlock better models on Amazon Bedrock

This article describes how Property Finder identified evaluation bias in their vision AI system and achieved better accuracy at lower cost by building unbiased ground truth and optimizing prompts on Amazon Bedrock.

• Discovered 96.8% of evaluation labels inherited from incumbent model predictions, creating circular evaluation bias
• Built unbiased ground truth with 356 expert-annotated images and switched to multi-label classification for overlapping attributes
• Applied evolutionary prompt optimization across 27 iterations, using error analysis and automated frameworks like MIPROv2
• Achieved 47% relative accuracy gain (Micro F1 from 0.55 to 0.81) on development set with Llama 4 Maverick 17B Instruct
• Reduced cost by 86% ($2.35 per 1,000 images) and inference latency by 40-60% compared to previous setup
• Processes tens of thousands of property images daily across MENA, evaluating 11 quality attributes per image

The methodology demonstrates how fixing evaluation methodology itself—not just finding better models—unlocks objective model comparison and significant cost and performance improvements.