Preparing data for supervised fine-tuning Part 2: Advanced data strategies

This article covers advanced data preparation strategies for supervised fine-tuning beyond basic formatting and quality checks, focusing on optimizing datasets for training efficiency and model performance.

• Use learning curve analysis to determine optimal dataset size by evaluating checkpoints and identifying saturation points where additional data yields diminishing returns.
• Apply data subset selection methods like DEITA and DELIFT to identify high-value examples, reducing dataset size while maintaining or improving performance.
• Augment datasets through distillation from stronger models, self-generation with filtering, or paraphrasing expert examples to expand coverage without proportional annotation costs.
• Mix target-task data with general instruction-following and safety data to preserve model capabilities, using batch-level composition with controlled proportions.
• Employ automated mixture optimization techniques like DoReMi and RegMix when managing multiple data sources to find near-optimal weights efficiently.
• Measure capability retention empirically through controlled experiments rather than applying default mixing ratios.

Effective data preparation requires treating optimization decisions as empirical questions, measuring performance at each stage to balance specialization with general capability preservation.