Accelerating log analytics at scale with AWS Glue and Apache Iceberg materialized views
Big Data Blog
This article demonstrates how to build a scalable log analytics pipeline using AWS Glue, Apache Iceberg materialized views, and Amazon Data Firehose to accelerate query performance on high-volume application logs.
- CloudWatch Logs routes application logs through Lambda for transformation, then to Firehose for delivery into Apache Iceberg tables on S3
- Materialized views pre-compute aggregations, enabling queries that previously took minutes to return in seconds
- AWS Glue scheduled jobs automatically refresh materialized views on configurable intervals to keep results current
- Solution provides ACID transactions, schema evolution, automatic scaling, and error handling with failed records routed to S3
- Deployment uses CloudFormation template to provision infrastructure; includes testing and monitoring through Amazon Athena
This serverless architecture minimizes operational overhead while delivering fast analytics on large-scale log data through pre-aggregated results.
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
Related articles
Dec 1
2025
2025
AWS Glue now supports Apache Iceberg based materialized views
Dec 9
2025
2025
Introducing Apache Iceberg materialized views in AWS Glue Data Catalog
Jul 1
2026
2026
Run log analytics for a fraction of the cost with the new engine for Amazon OpenSearch Service
Jul 9
2024
2024
Accelerate query performance with Apache Iceberg statistics on the AWS Glue Data Catalog
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.