Llama 3.1 405B, an open-source AI model with 405 billion parameters, is the largest model that Meta has released so far. It's not the largest model overall, but it is the largest open-source model that has been made available recently. Utilizing new methodologies, Llama 3.1 405B, trained on 16,000 Nvidia H100 GPUs, can rival top-tier proprietary models like Anthropic's Claude 3.5 Sonnet and OpenAI's GPT-4o. The model may be downloaded and used on cloud computing platforms including Google Cloud, AWS, and Azure. It is also integrated into chatbots in the United States using WhatsApp and Meta.ai.
Though it is restricted to text-based activities, Llama 3.1 405B can execute a wide range of tasks, including coding and multilingual document summarizing (in English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai). While these are not currently publicly accessible, Meta is actively working on multimodal Llama models to handle photos, videos, and speech. The model was reinforced with synthetic data, which is usual but raises questions about potential bias, then trained on a refined sample of 15 trillion tokens.
The model can effectively handle longer inputs and keep conversation context thanks to its 128,000 token context window. The Llama 3.1 8B and Llama 3.1 70B are smaller models that can also utilize third-party tools and APIs to increase their adaptability in addition to having this expanded context capability. These models can communicate with a Python interpreter for code checking, Brave Search, and Wolfram Alpha for mathematical questions.
Meta wants to create a developer ecosystem around Llama so that it becomes a pillar of generative AI. With some limitations on deployment for larger developers, the new license permits developers to use model outputs for developing third-party generative models. In order to facilitate fine-tuning, producing synthetic data, and developing sophisticated applications, Meta is also releasing new safety tools and the Llama Stack API.
CEO Mark Zuckerberg places a strong emphasis on democratizing access to AI while establishing Meta's AI technologies as industry norms. Over 300 million Llama models have been downloaded, and 20,000 variant models have been made, despite legal issues and worries over data practices.
Energy difficulties arise when scaling these models since training puts a strain on power grids. Since Meta intends to build even larger models in the future, resolving these difficulties will be essential. With the release of Llama 3.1 405B, Meta has taken a significant step toward its AI strategy, which aims to push the boundaries of generative AI and challenge competitors.