Use AWS Glue to streamline SFTP data processing
Big Data Blog
This article explains how to use the SFTP Connector for AWS Glue to streamline the processing of data from SFTP servers into Amazon S3. It demonstrates the steps to connect to an SFTP server using AWS Glue, extract data from the SFTP server, join it with data from Amazon S3, transform the data, and load the transformed data into Amazon S3.
Specifically, the article covers:
- Introduction to the SFTP Connector for AWS Glue
- Solution overview and architecture
- Prerequisites (subscribing to the SFTP Connector, access to SFTP server and S3 bucket, creating IAM role)
- Storing SFTP server credentials in AWS Secrets Manager
- Creating connections in AWS Glue (to SFTP server and VPC)
- Deploying the solution by creating a Glue Studio job, configuring data sources, transformations, and loading data to S3
- Cleaning up resources created for the solution
- Conclusion
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
Related articles
2024
2024
2026
2025
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.