Fine-tune LLM with Databricks Unity Catalog and Amazon SageMaker AI

This article demonstrates how to build a secure LLM fine-tuning workflow integrating Databricks Unity Catalog with Amazon SageMaker AI, maintaining data governance across services.

• Reads training data from Unity Catalog-managed tables with governance controls
• Preprocesses data using EMR Serverless with Apache Spark
• Fine-tunes Ministral-3-3B-Instruct model using SageMaker AI Training jobs
• Tracks complete data lineage from source data through trained model
• Stores Databricks OAuth credentials securely in AWS Secrets Manager
• Uses External Lineage API to capture lineage for non-Databricks workloads
• Registers fine-tuned models back into Unity Catalog with MLflow
• Provides complete notebook implementation and step-by-step walkthrough

This integration enables governed, production-ready LLM fine-tuning while maintaining compliance, audit trails, and centralized access control across AWS and Databricks services.