Inside Meta’s Llama 3 Journey: From Data Curation to Inference
Meta’s Llama 3 is the company’s most capable openly available LLM to date, and the recently released Llama 3.1 expands its utility to new workflows like synthetic data generation and model distillation. According to Meta, these capabilities come with state-of-the-art performance that competes with the best closed-source models, while offering developers greater flexibility and control.
At AI Infra @ Scale 2024, Meta engineers peeled back the curtain on the full lifecycle of the model, covering data, training, and inference. The session kicked off with Joe Spisak, Product Director and Head of Generative AI Open Source at Meta, who traced the history of the Llama family and outlined the company’s broader vision for open-source AI.
Spisak was joined by software engineer Delia David, who dove into the data side of GenAI. David highlighted the diversity, volume, and freshness required for training, as well as best practices for extracting and preparing different data types.
Next, software engineer Kaushik Veeraraghavan explained how Meta trains Llama at scale, detailing the data center, networking, and software investments that enabled the development of the Llama 3 models. Finally, production engineer Ye (Charlotte) Qi discussed inference handling, focusing on parallelism techniques that help scale both model sizes and context windows. Qi also addressed the practical challenges of deploying these complex serving architectures across Meta’s internal cloud, which relies on heterogeneous hardware.
