Multimodal embeddings at scale: AI data lake for media and entertainment workloads

This article demonstrates building a scalable multimodal video search system using Amazon Nova embeddings and OpenSearch Service for semantic search across large video datasets.

• Processed 792,270 videos (8,480 hours) in 41 hours; first-year cost: $27,328 on-demand or $23,632 with Reserved Instances
• Uses Nova Multimodal Embeddings to generate audio-visual embeddings in AUDIO_VIDEO_COMBINED mode
• Videos segmented into 15-second chunks; 1024-dimensional embeddings chosen for 3x cost savings
• Dual OpenSearch indexes: k-NN for semantic search, text index for keyword matching
• Three search modes: text-to-video (natural language), video-to-video (similarity), hybrid (70% vector/30% keyword)
• Search latencies at 792K videos: semantic ~76ms, BM25 ~30ms, hybrid ~106ms
• Total storage: 29.8 GB (k-NN: 28.8 GB, text: 1.0 GB)
• Nova Pro generates 10-15 descriptive tags per video from predefined taxonomy
• Async API manages 30 concurrent job limit with polling queue system
• Ingestion cost breakdown: EC2 $421, Nova embeddings $17,096, Nova Pro tagging $571

The solution enables semantic video search at scale with sub-200ms query latencies, combining audio-visual embeddings with keyword matching for production media and entertainment workloads.