How CFM built a well-governed and scalable data-engineering platform using Amazon EMR for financial features generation

This article discusses how Capital Fund Management (CFM), an alternative investment management company, built a well-governed and scalable data engineering platform on AWS for generating financial features to drive their investment strategies.

Specifically, the article covers:

• CFM's strategy to leverage alternative data sources and innovation to stay competitive in quantitative investing
• How AWS enables this data strategy through managed services, elasticity, and governance capabilities
• CFM's data integration workflow involving data ingestion, analysis with Jupyter notebooks and Amazon EMR, and production deployment
• The solution architecture integrating Apache Livy, Sparkmagic, and Amazon EMR to enable interactive data exploration from Jupyter notebooks
• Using Amazon EMR Serverless in production to reduce startup times and operational overhead
• Implementing a data mesh architecture with AWS Lake Formation for cross-account data sharing and governance
• Conclusion highlighting the benefits of using Lake Formation and EMR Serverless for streamlining analytics workflows