Faster LLMs with speculative decoding and AWS Inferentia2

This article discusses a technique called "speculative sampling" that can improve the computational efficiency and performance of large language models (LLMs) like Llama-3-70B and Llama-2-70B when running inference on AWS Inferentia and Trainium chips.

Specifically, the article covers:

• Introduction to sequential token generation in LLMs and its computational challenges
• How speculative sampling works by using a smaller "draft" model to speculate tokens that are verified by a larger "target" model
• A walkthrough of using speculative sampling with Llama-2-70B and Llama-2-7B models on Inferentia2 and Trainium instances
• Considerations for loading models, memory requirements, and tensor parallelism
• Conclusion on how speculative sampling can enable using larger, higher-quality LLMs while maintaining speed and cost-efficiency