A group of former MIT researchers has launched Liquid AI, introducing a fresh lineup of AI models that steer away from the transformer architecture used in most modern AI systems. The new models, called Liquid Foundation Models (LFMs), have been designed to handle various tasks with less memory consumption, all while performing better than existing transformer-based models from major players like Meta and Microsoft.
Liquid AI's move away from the transformer framework, which has dominated AI since the release of Attention Is All You Need in 2017, puts them in a strong position in the growing field of AI innovation. The company has unveiled three models: LFM-1.3B, LFM-3B, and LFM-40B MoE. Each number in their names reflect the billions of parameters that define how each model processes and outputs data.
Better Memory Efficiency and Handling Long Inputs
Their top-tier model, LFM-40B, incorporates a Mixture of Experts architecture that activates 12 billion parameters during use. This approach allows it to deliver impressive performance while maintaining efficiency, often surpassing traditional transformer models in head-to-head comparisons.
Liquid's models stand out primarily due to their ability to manage memory more effectively. Drawing on principles from signal processing and other mathematical theories, these models avoid the sharp increase in memory usage that transformer-based systems typically experience when dealing with longer input sequences.
For example, Liquid's models can process up to 32,000 tokens without requiring significantly more memory, which is a game-changer for applications like document processing or chatbot systems that need to handle long conversations or large chunks of text.
Maxime Labonne, the Head of Post-Training at Liquid AI, highlighted the models' ability to outperform transformer-based systems with much lower memory demands, making them ideal for scenarios where memory resources are limited, such as mobile or edge deployments.
A Fresh Approach to AI Model Design
Instead of relying on the widely used GPT framework that underpins many AI models, Liquid AI chose a different direction. By building on decades of research in mathematical fields, the company developed a model capable of working with a broad range of data types, including text, video, and audio. The flexibility means the models can be applied to industries as diverse as healthcare, finance, and consumer technology.
Liquid's models are currently in the preview phase, but early interest from enterprise users has been strong. The company is refining its offerings based on user feedback as it prepares for a public event on October 23, 2024, at MIT. Liquid has encouraged developers and businesses to experiment with the models through its online platforms, including Liquid's inference playground.
Future Plans and Challenges
Despite these advances, Liquid AI admits there are still challenges to overcome. For instance, its models struggle with certain tasks, such as zero-shot code generation and complex numerical operations. Despite outperforming other models, these limitations show that while Liquid's approach offers promise, the technology is still evolving.
The models' efficient use of memory stands out, particularly with the LFM-3B model, which requires just 16 GB of memory—far less than many models of similar size. This reduced demand opens the door for more resource-constrained applications, from mobile devices to edge computing.
Looking ahead, Liquid AI is working to make its models more accessible by optimizing them for a variety of hardware platforms. The company is collaborating with major chipmakers like NVIDIA, AMD, Qualcomm, Apple, and Cerebras to ensure its models can run efficiently across different types of processors.
Last Updated on October 14, 2024 12:43 pm CEST
