Accelerating LLM fine-tuning with unstructured data using SageMaker Unified Studio and S3

This article demonstrates how to fine-tune Llama 3.2 11B Vision Instruct for visual question answering using SageMaker Unified Studio and S3 integration.

• Integrates S3 general purpose buckets with SageMaker Catalog for unstructured data access
• Uses DocVQA dataset with three training sizes (1k, 5k, 10k images) for fine-tuning
• Data producer project catalogs S3 data; consumer project subscribes and develops models
• Fine-tuned model achieves 0.902 ANLS score, 4.9% improvement over base model (0.853)
• MLflow tracks experimentation and evaluation metrics across all training runs
• S3 Access Grants provide secure data access without complex permission management
• Requires ml.p4de.24xlarge instance type; training takes approximately 4 hours per model

This integration streamlines ML workflows by enabling secure data discovery, cataloging, and collaboration between data producers and consumers while maintaining governance controls.