Building an End-to-End Physical AI Data Pipeline for Autonomous Vehicle 3.0 on AWS with NVIDIA

This article presents a reference architecture for building Autonomous Vehicle 3.0 (AV 3.0) data pipelines on AWS with NVIDIA, spanning fleet sensor ingestion through model validation.

• AV 3.0 uses end-to-end Vision-Language-Action systems requiring vast real-world and synthetic sensor data
• Eight-stage pipeline: ingest raw fleet data, validate quality, curate with AI, search/index scenarios, augment data, reconstruct 3D scenes, train reasoning VLA models, validate in simulation
• Fleet sensor data shipped to AWS Data Transfer Terminal, stored in Amazon S3 with intelligent tiering
• NVIDIA Cosmos Curator generates semantic captions and embeddings for video curation on SageMaker HyperPod
• Dual search paths: Amazon OpenSearch Service with GPU acceleration or NVIDIA Cosmos Dataset Search on EKS
• NVIDIA Cosmos Transfer generates photorealistic synthetic variants for data augmentation
• NVIDIA Omniverse NuRec reconstructs real-world sensor data into drivable 3D scenes
• NVIDIA Alpamayo trains unified perception-reasoning-action models on curated gold datasets
• NVIDIA AlpaSim enables closed-loop simulation validation with physics and metrics collection
• Iterative feedback loop: curate → reconstruct → train → validate → repeat for rapid hypothesis testing
• Architecture is modular; stages can be adopted independently and integrated with existing systems

This end-to-end pipeline enables autonomous vehicle teams to scale development through AI-enhanced data curation, retrieval-driven dataset assembly, and fast closed-loop validation on managed AWS infrastructure.