Governing streaming data in Amazon DataZone with the Data Solutions Framework on AWS

This article discusses how to extend Amazon DataZone to govern streaming data, specifically using Amazon Managed Streaming for Apache Kafka (MSK) as an example. The solution addresses the challenges of managing real-time data streams within a comprehensive data governance framework.

• Creates custom asset types in Amazon DataZone to represent Kafka topics
• Develops a custom data source using AWS Lambda to import MSK metadata from AWS Glue Schema Registry
• Implements a custom authorization flow for streaming data subscription and access management
• Uses the Data Solutions Framework (DSF) on AWS to accelerate implementation
• Provides a comprehensive approach to discover, catalog, and securely share streaming data assets

The solution enables organizations to extend data governance practices to streaming data, making it discoverable, secure, and manageable across different environments and accounts.