Build a real-time event pipeline with Spark Real-Time Mode on AWS Glue 6.0

This article demonstrates how to build a multi-layer real-time event pipeline using AWS Glue 6.0 for processing trades with heterogeneous schemas across real-time, near-real-time, and batch layers.

• Spark Real-Time Mode eliminates micro-batch boundaries for sub-second trade scoring and position breach alerts
• Iceberg v3 Variant columns store semi-structured pricing vectors without requiring fixed schemas upfront
• Variant shredding automatically extracts fields into typed Parquet columns for faster downstream queries
• Arrow-native UDFs process data as vectorized columnar batches, eliminating pickle serialization overhead for Python risk calculations
• Deletion vectors enable efficient row-level updates and cancellations using compact Roaring Bitmaps instead of file rewrites
• DEFAULT values reduce boilerplate in ingestion code for missing fields
• Solution uses three independent Glue jobs: real-time alerts (Scala), near-real-time ingestion (PySpark), and batch VaR analytics (PySpark)

The architecture handles schema heterogeneity, JSON parsing overhead, and mixed-latency requirements in a single unified pipeline applicable to IoT, SaaS, and logistics use cases.