Build efficient agents with knowledge distillation on Amazon SageMaker AI

This article demonstrates how to use sequence-level knowledge distillation (SeqKD) to fine-tune a smaller shopping agent model on Amazon SageMaker AI, achieving frontier-model quality at reduced cost and latency.

• SeqKD transfers behavior by training a student model on teacher-generated trajectories without requiring access to teacher weights or logits
• Generate high-quality multi-step trajectories with a frontier teacher and filter using a programmatic scorer to keep only solved runs
• Encode trajectories as chat-formatted messages preserving reasoning alongside tool calls for supervised fine-tuning
• Fine-tune a Qwen3.5-4B student serverlessly using SageMaker AI model customization with LoRA adaptation
• Distilled 4B student achieves 54.1% success rate, matching the frontier teacher's 51-54% across shopping tasks
• Serve the student as a LoRA adapter on a shared base model endpoint for cost-effective inference
• Reasoning in training targets carries most of the performance gain, adding ~13 points over tool-calls-only training

SeqKD enables cost-effective agentic applications by distilling frontier model behavior into smaller deployable models, working with API-only teachers and eliminating per-token billing after deployment.