Large language models are not disappearing, but the machinery inside them may be due for a rethink. Since the summer of 2017, when AI researchers at Google published “Attention Is All You Need,” transformers have become the foundation of every major LLM on the market.
That dominance is now under pressure. Newer models need to handle bigger inputs, support reasoning-style workflows and consume less computing power. A group of startups is betting that the next phase of LLMs will come from changing, shrinking or replacing key parts of the transformer stack.
Why Transformers Became The Bottleneck
The core advantage of transformers is dense attention. This mechanism encodes the meaning of text as numbers by comparing every word, or token, with every other token in a block of text.
That approach is powerful, but it becomes expensive as input grows. A document 10,000 words long might require 50 million multiplications. That computation load is one reason LLMs use so much power.
The costs around this model are already large. OpenAI is set to spend $50 billion on computing this year, according to the company’s president, Greg Brockman. The International Energy Agency also predicts that total electricity consumed by data centers will double by 2030.
The technical issue is not only cost. Transformers process text word by word, which makes it hard to track very large amounts of information at once. Yet harder tasks may require models to take in a whole library of documents, an entire code base or output from other LLMs.
Reasoning models add another pressure point. They work by producing notes to themselves, known as a chain of thought, and then reading those notes back. That gives the model more material to manage, which makes the transformer’s strength start to look like a constraint.
Changing Attention Instead Of Abandoning It
Subquadratic, based in Miami, is trying to solve the problem by changing attention directly. Its approach uses sparse attention, which calculates only some relationships between words instead of comparing every word with every other word.
Sparse attention has been explored for years, but the problem has been quality. Earlier versions did not capture meaning as well as dense attention. Subquadratic claims its model, SubQ, can rival top mainstream LLMs on tasks including search and coding, although some people in the industry remain skeptical.
SubQ is designed to decide, for each input, which words matter and which do not. The company says thousands have joined its waitlist and plans to make the model widely available soon.
Manifest AI, based in San Francisco, is taking a related but different path. Rather than altering attention, it replaces it with a mechanism called power retention.
Power retention keeps only the information that is most relevant for a task. Instead of holding everything in the context window, it builds a rolling summary and drops less relevant information as new information arrives.
The basic idea of retention has been around for a decade. Manifest AI says it has updated those methods enough to compete with transformer-based LLMs. The company also says an existing transformer model can be adapted into a power retention model with minimal retraining.
To show this, Manifest AI turned StarCoder, an open-source coding LLM, into PowerCoder. It has also released Brumby, which it claims rivals some versions of Alibaba’s open-source model Qwen.
Smaller Models And New Neural Building Blocks
Liquid AI, an MIT spinout based in Cambridge, Massachusetts, is not fully discarding transformers. Instead, it combines them with liquid neural networks to create what cofounder and CEO Ramin Hasani calls LFMs, or liquid foundation models.
Liquid AI focuses on models that are smaller and use less energy than most LLMs. It builds models for car makers, including Mercedes, that run on small chips inside vehicles. Its latest models can run on a Raspberry Pi, a low-powered hobbyist computer that costs $50.
The company makes its models available for free to any organization with annual revenue below $10 million. Hasani says the models have reached almost 34 million downloads.
Liquid neural networks were inspired by worm brains and extend convolutional networks, a type of neural network that predates transformers. Their key feature is the ability to adapt behavior when new information arrives, so the model can learn as it goes. A transformer’s behavior is fixed after training.
Liquid AI’s early models could fly drones or drive vehicles. With LFMs, the company is trying to scale the technology toward mainstream LLM performance. Its newer models match rivals four times bigger, including versions of Alibaba’s Qwen and Google’s open-source LLM Gemma.
The company’s recent LFMs are hybrids: 20% transformers and 80% liquid neural networks. That ratio came from another AI system Liquid AI uses to design its models by testing combinations of liquid, convolutional and transformer components.
Generating Text Differently
Most LLMs generate output one word at a time. Inception, based in Palo Alto, California, is trying to make that process faster by using diffusion, a technique better known from image and video generation models.
Diffusion models for images start with a random grid of pixels and gradually turn it into an image. Inception applies a related idea to text by training LLMs to turn a random string of words into sentences that make sense.
These diffusion LLMs still use transformers to encode meaning, but they generate blocks of text at once. That lets the transformer do more work with fewer steps.
The hard part was adapting diffusion from pixels to words. Stefano Ermon, Inception’s cofounder and CEO and a researcher at Stanford University, worked with two Stanford colleagues in 2024 on the math needed to make diffusion function for text.
They trained a diffusion model that matched GPT-2, an OpenAI LLM from 2019, while running 10 times faster. Inception now says its latest model, Mercury 2, performs as well as some of OpenAI’s GPT-4 models released in 2023, while again being 10 times faster.
Beyond Word-By-Word Reasoning
Pathway, also based in Palo Alto, is making one of the sharpest breaks from the usual LLM design. Its model, Dragon Hatchling, is built around the idea that some reasoning should not be forced into language.
Dragon Hatchling’s standout result is on a benchmark of more than 250,000 very hard sudoku puzzles. It beat more than 97% of the puzzles, while several leading LLMs from top labs failed to solve any.
Pathway’s cofounder and CEO, Zuzanna Stamirowska, argues that sudoku shows a broader weakness. Some problems are difficult to represent word by word, and language may not be the best format for reasoning about them.
Pathway replaces the attention mechanism with a mathematical structure called a state space. Instead of encoding information as a sequence of words, state spaces compress it into a more abstract representation.
That still lets Dragon Hatchling process and produce text. In theory, it also lets the model work on tasks that do not fit neatly into language, such as chess or mathematics.
The larger point is that transformer-based LLMs may not be the final form of AI models. The next generation could keep parts of today’s systems while changing attention, retaining less data, combining neural network types, generating text in blocks or reasoning in more abstract ways.