Training Azerbaijani language models on Amazon SageMaker AI

This article describes how Azercell Telecom built an Azerbaijani language model on Amazon SageMaker AI through a three-stage framework combining custom tokenization, continued pre-training, and fine-tuning.

• Custom tokenizer reduced tokens per word by 50%, doubling context window capacity
• FSDP and Liger Kernel optimizations achieved 58% lower peak GPU memory usage
• Training throughput increased 23% on ml.p5.48xlarge instances
• Continued pre-training adapted Llama 3.2 1B to understand Azerbaijani morphology
• LoRA fine-tuning with 2,000 question-answer pairs created conversational assistant
• Framework scales from 1B to larger models with configuration changes only

The solution demonstrates efficient LLM adaptation for morphologically rich, low-resource languages using AWS infrastructure and open-source tools.