How Common Crawl and AWS Open Data built the foundation for the AI revolution

This article describes how Common Crawl and AWS Open Data built the foundation for the AI revolution by providing free access to web-scale training data for large language models.

• Common Crawl has hosted 300 billion web pages on AWS S3 since 2012 at no cost through the AWS Open Data Sponsorship Program
• 64% of major LLMs published 2019–2023 used filtered Common Crawl data for training, cited in 13,000+ research papers
• Infrastructure evolved to include S3, CloudFront, Athena, EMR, Lambda, and SageMaker for global access and processing
• Enables researchers, builders, and organizations worldwide to access foundational AI training data without building crawling infrastructure
• Web Languages Project and CommonLID benchmark address multilingual AI representation across 109+ languages

Open data infrastructure has become essential to AI development, enabling researchers globally to access the same foundational datasets powering frontier AI systems.