Home icon

How Pariveda Enables Operational Data Observability Across Your AWS Data Lake at Scale

AWS Partner Network Blog

This article discusses how Pariveda, an AWS Premier Tier Services Partner, enables operational data observability across AWS data lakes at scale. It presents a solution to collect and visualize AWS Glue job metrics in a QuickSight dashboard, allowing for monitoring and optimization of ETL workloads.

Specifically, the article covers:

  • Customer requirements for a healthcare client to improve visibility into AWS Glue job performance
  • The solution architecture leveraging AWS services like CloudWatch, S3, Athena, QuickSight, and AWS Glue Data Catalog
  • Core solution workflow to stream AWS Glue metrics to S3 and create a QuickSight dashboard
  • Access management using AWS Lake Formation for centralized permissions
  • Capturing custom metrics like AWS Glue JobRunId using AWS Lambda
  • Examples of the operational QuickSight dashboard visualizations
  • Conclusion highlighting the benefits of the solution for optimizing data pipelines
Go to article

The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.

Related articles

Feb 15
2024
Achieve Complete Data Observability on AWS with TCS Approach to Observability Challenges
Apr 17
2024
Open source observability for AWS Inferentia nodes within Amazon EKS clusters
Apr 16
2024
End-to-End Observability for SAP on AWS: Part 2 – SAP Network Latency Monitoring
Apr 17
2024
Accelerating data ingestion and on-boarding with new AWS Supply Chain capabilities

The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.