Preparing data for supervised fine-tuning Part 1: Formatting and quality

This article provides foundational guidance on preparing high-quality data for supervised fine-tuning (SFT) of foundation models, covering quality checks, formatting requirements, and train/evaluation splits.

• Implement rigorous quality checks: ensure accuracy, diversity of examples, consistency within task types, deduplication, and safety screening before training
• Use conversational JSONL format with strict role alternation between user and assistant turns, matching the model's original chat template exactly
• Include system prompts in training data when they'll be used during inference to avoid distribution mismatch
• For reasoning-enabled models, include intermediate thinking steps using reasoningContent field to transfer chain-of-thought behavior
• Support tool calling and multimodal formats with proper toolUse/toolResult blocks and S3-referenced documents or images
• Hold out 10-20% of data as a representative evaluation set with stratified splitting for small datasets
• Quality beats quantity: carefully curated examples outperform larger datasets with lower quality

Proper data preparation is the most impactful activity in SFT workflows, enabling faster convergence, better generalization, and avoiding costly debugging cycles.