Scaling Thomson Reuters’ language model research with Amazon SageMaker HyperPod

This article describes how Thomson Reuters scaled their research on training large language models (LLMs) using Amazon SageMaker HyperPod, a service that provides resilient and persistent clusters for distributed deep learning training.

Specifically, the article covers:

• The motivation for Thomson Reuters to train domain-adapted LLMs, including the limitations of public/commercial models like hallucinations, lack of domain knowledge, and speed/cost
• Thomson Reuters' research objectives around improving model performance on legal tasks using domain data and different training techniques like continuous pre-training and instruction fine-tuning
• Details on scaling model training using Amazon SageMaker HyperPod, including resilience features like deep health checks, automatic node replacement, and auto-resume
• Initial findings showing improvements in legal summarization and classification tasks with domain-adapted models compared to GPT-4
• Conclusion highlighting the benefits of Amazon SageMaker HyperPod for LLM training at scale