LLM continuous self-instruct fine-tuning framework powered by a compound AI system on Amazon SageMaker

This article introduces a continuous self-instruct fine-tuning framework for large language models (LLMs) using a compound AI system on Amazon SageMaker. The framework addresses challenges in LLM customization and performance improvement through several key approaches:

• Self-instruct supervised fine-tuning to synthetically generate training labels
• Human preference alignment using reinforcement learning techniques
• Continuous evaluation and learning to monitor model performance
• Using DSPy framework to build a modular, flexible AI system

The authors demonstrated the framework using a question-answering task with a Retrieval Augmented Generation (RAG) pipeline, showing significant accuracy improvements:

• RAG optimization improved accuracy by 67-110%
• Supervised fine-tuning increased accuracy by 21-59%
• Preference alignment methods (DPO/ORPO) boosted accuracy by 69-125%

The compound AI system enables modular design, increased flexibility, and adaptability for continuous LLM performance improvement.