LLM experimentation at scale using Amazon SageMaker Pipelines and MLflow

Machine Learning Blog

This article explains how to run experiments for fine-tuning large language models (LLMs) at scale using Amazon SageMaker Pipelines and MLflow. It covers:

Setting up an MLflow tracking server on SageMaker Studio
Logging datasets with MLflow for tracking and reproducibility
Fine-tuning an LLM like Llama using Low-Rank Adaptation (LoRA) and logging hyperparameters and model with MLflow
Evaluating the fine-tuned model using MLflow's evaluation capabilities
Creating and running a SageMaker Pipeline for orchestrating the fine-tuning and evaluation experiments
Comparing results across different experiment runs using the MLflow UI
Registering the best model with MLflow and deploying it on SageMaker
Cleaning up resources after the experiments

Go to article

The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.

Jan 28
2025

Track LLM model evaluation using Amazon SageMaker managed MLflow and FMEval

Apr 22
2025

Supercharge your LLM performance with Amazon SageMaker Large Model Inference container v15

Apr 24
2024

Improve LLM performance with human and AI feedback on Amazon SageMaker for Amazon Engineering

Mar 26
2026

Accelerating LLM fine-tuning with unstructured data using SageMaker Unified Studio and S3

The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.

LLM experimentation at scale using Amazon SageMaker Pipelines and MLflow

Related articles