Distillation: Making AI Models More Efficient and Affordable
Originally Published 5 months ago — by Quanta Magazine
DeepSeek's use of knowledge distillation, a widely used AI technique that involves training smaller models using the outputs of larger ones, has sparked controversy but is a common practice in AI development. Originally developed in 2015 at Google to make ensemble models more efficient, distillation helps create smaller, cheaper, and faster AI models by transferring 'dark knowledge' from a teacher to a student model. It has become a fundamental tool in AI, enabling companies like Google, OpenAI, and Amazon to deploy powerful models more efficiently, and continues to be an active area of research and application.