Amazon’s renewable energy forecasting: continuous delivery with Jupyter Notebooks

The article discusses how Amazon's Renewable Energy Optimization team transitioned to using Jupyter Notebooks in production for their machine learning models that forecast electricity grid states for Amazon's wind and solar farms.

Specifically, the article covers:

• The initial challenges of having scientists develop models in Jupyter Notebooks and engineers translating them to production code, leading to delays, lack of understanding, and inconsistencies between environments.
• How the Papermill library allowed running Jupyter Notebooks programmatically as immutable objects, enabling a continuous delivery workflow.
• Using SageMaker custom images to ensure the same environment and dependencies between development and production.
• The production setup involving AWS Batch, Amazon ECS, EventBridge for scheduling, and infrastructure provisioned via AWS CDK.
• The benefits achieved including increased velocity, reliability, and collaboration between scientists and engineers.