Meta Reveals Flame 3: We’re Testing a New Open Source AI Model

Meta has released Llama 3, the most advanced open source large language model (LLM) currently available. It builds on the foundation laid by its predecessor, Llama 2, and took everyone by surprise given rumors that it would release next month.

Being open source, Llama-2 played an important role in the simultaneous development of other powerful models such as Mixtral, Alpaca, Vicuna and WizardLM. Now Llama 3 promises to expand on these capabilities, offering functionality comparable to OpenAI’s current flagship AI model, GPT-4.

Meta hailed Thursday’s release as “the next generation of our cutting-edge, large-scale, open-source language model.” The tech giant is so confident in its capabilities that Llama 3 is used in the company’s AI assistant Meta AI, which in turn has been added to almost all of the company’s popular apps: Instagram, Facebook and WhatsApp. It is available in some countries, but users in other regions can access it through a VPN.

Meta AI Chatbot’s interface is similar to ChatGPT Plus, and it’s free!

“We are updating Meta AI with our new next generation AI model, Llama 3, which we are publicly sharing,” Mark Zuckerberg said in a Facebook post. “We believe that with this new model, Meta AI has become the smartest AI assistant that you can freely use.”

Decipher was able to test the new AI and found it to be just as good as ChatGPT-Plus, but without the need for a paid subscription. It can generate images and animations, generate code, and provide consistent and contextually relevant responses. The new chatbot can also access the Internet, but it still doesn’t compare to the capabilities of dedicated solutions like Perplexity.

Perhaps the only downside is that the current Llama-3 context window is limited to 8,000 tokens, or about 6,000 words.

Meta has released a Llama-3 model with 70 billion parameters, but using it will require a lot of computing power, perhaps an entire rack of GPUs. In synthetic tests, this model outperforms Gemini 1.5 Pro and Claude 3 Sonnet.

An 8 billion parameter model is also available that can run locally on consumer GPUs. It beats Google Gemini and Mistral 7B in several synthetic benchmarks. The model has not yet been exhibited on the LLM Arena, so there are no subjective ELO ratings yet.

Image: Meta

Both models can also run in the cloud at a lower cost.

“We are committed to responsible development of Llama 3 and offer several resources to help others use it responsibly,” Mehta said. This includes the introduction of new trust and security tools such as Llama Guard 2, Code Shield and CyberSec Eval 2.

In the coming months, Meta says it plans to introduce new features, longer context windows, additional model sizes, and improved performance. A research paper on Llama 3 will also be published.

“Meta AI, powered by Llama 3 technology, is now one of the world’s leading AI assistants that can boost your intelligence and lighten your load, helping you learn, get things done, create content and communicate to get the most out of it from every moment,” Mehta said.

Meta added that it is also training a huge model with 400 billion parameters, which is expected to be released later this year. This model, likely comparable to Claude Opus or the latest GPT-4.5, could be the most powerful open source model to date. If history repeats itself, it will also serve as the basis for a new generation of improved models that will surpass Llama 3 in overall quality and increase competition from leading closed-source models.

Testing Llama 3

Decipher tested Llama 3 inside Meta AI to make sure it’s as good as Zuck says. To sum it up, Llama-3 introduced a number of notable features and capabilities and should be an excellent baseline model for the open source community.

Content moderation

Llama 3 demonstrates a strong commitment to content moderation. He has consistently refused to create harmful racial content, even when faced with common hacking techniques.

For example, when the model was asked to give instructions on how to seduce a woman, he gave general but useful answers. However, when asked how to seduce the wife of her best friend, the model categorically refused to answer.

Images and animation

Similar to ChatGPT-Plus, Meta AI with Llama-3 is capable of generating images. However, it takes this feature further by offering the ability to animate them, a feature not available in ChatGPT or Gemini.

Images created by Meta AI using Llama-3 are more realistic than images created by Dalle-3, but do not reach the quality of images created by ImageFX, Google’s upcoming tool.

Encoding capabilities

Lama 3 has proven that she can program very well. When presented with a unique and poorly explained game idea, the model was able to generate the required Python code in two tries, resulting in a working game. The first attempt gave us a rough idea of ​​how to build the game, but generated working code after we specified that we needed it in Python.

The game worked, but some small details were missing, such as restarting after a player wins. However, the same thing happened with other chatbots.

We found Claude 3 Sonnet to be the best tool for this task, followed by Llama 3. GPT-4 drops to third place. Although different users may have different results.

Here is a pastabine (copy) with source codes generated by Llama3, Claude and ChatGPT for those who want to try them out.

Political neutrality

The model aims to be politically neutral, as evidenced by its answers to questions about capitalism and communism. The answers were structurally similar and provided an introduction, pros and cons of each system.

This pattern of neutrality was also observed in responses to questions such as “What is a man?” and “What is a woman?”

However, their responses are slightly pro-capitalist and left-wing, which is not surprising since this is the most common political tendency among the major language models.

Logical reasoning

Lama 3 demonstrated powerful logical reasoning abilities. When tested with difficult LSAT (Law School Admissions Test) questions that often confuse users, the model not only provided correct answers, but also offered clear and reasonable explanations.

Long indication limits

Despite its many strengths, Lama 3 doesn’t handle long clues well. When presented with a long hint of approximately one and a half pages of context that could be learned by models such as GPT-4, Claude or Mistral, the model returned an error message.

Understanding the language

The model demonstrates good understanding of different languages. When asked to translate the slogan into Spanish, he not only provided an accurate translation, but also offered context to better understand the slogan.

Conclusion

As a chatbot interface, Meta AI (powered by Llama3) can compete with ChatGPT Plus and is an excellent choice.

On a more technical level, Llama 3 as an LLM is good enough to compete with GPT-4 in a variety of scenarios, losing only in terms of token context capabilities and advanced recovery generation (essentially extracting information from a specific user-supplied data set). This may be important for technical users, but may not be so relevant for the average person.

If you primarily use ChatGPT to generate images with Dall-E, you may want to consider unsubscribing as Llama-3’s image generation and animation capabilities are comparable. However, if you need support for long instructions, Llama 3 may not be the best option for you and you may want to stick with ChatGPT-Plus.

Casual users may find that Llama 3 meets their needs without the need for a paid membership.

For tasks that require intensive Internet searching, ChatGPT Plus or Perplexity are more suitable.

Finally, if your focus is on programming, Llama 3 may be a good alternative, although other specialized tools are available. The fact that Llama-3 is free is a significant advantage.

Edited by Ryan Ozawa.