Databricks is putting a new marker down in the race to build powerful open source AI. After months of work and about $10 million in training costs, the company says its DBRX large language model has surpassed leading open source rivals and, on several scores, came unexpectedly close to GPT-4.
DBRX enters a crowded AI race
The final assessment of DBRX came after a high-pressure internal review involving about a dozen engineers and executives at Databricks. The model had been trained for months, but the team did not know how it would compare until the last test results arrived.
Jonathan Frankle, chief neural network architect at Databricks and leader of the DBRX team, presented results showing that DBRX performed better than every other open source model available across about a dozen or so benchmarks. Those tests measured abilities such as answering general knowledge questions, reading comprehension, solving difficult logical puzzles, and producing high-quality code.
The comparison set included Meta’s Llama 2 and Mistral’s Mixtral, both described in the source as among the most popular open source AI models available today. DBRX also surpassed the Grok AI model recently open-sourced by Musk’s xAI.
Databricks says the model was also surprisingly close to GPT-4 on several scores. GPT-4 remains a closed model from OpenAI and powers ChatGPT, which the source describes as widely considered the pinnacle of machine intelligence.
Why open source matters here
Databricks plans to release DBRX under an open source license. That matters because it gives researchers, entrepreneurs, startups, and established businesses a foundation they can build on rather than a system they can only access through a closed provider.
The move also adds weight to a broader split in generative AI. OpenAI and Google keep the code for GPT-4 and Gemini closely held. Other companies, notably Meta, have released models for others to use, arguing that wider access can accelerate innovation.
Databricks is also emphasizing transparency around the work of building DBRX. The company plans to publish a blog post with details about the process and gave WIRED access to engineers during the final stages of training. According to the source, that access offered a view into how complex the process is, and how recent advances may reduce the cost of creating strong AI models.
Ali Farhadi, CEO of the Allen Institute for AI, said greater openness is needed because the field has become more secretive as companies compete for an advantage. He also connected that opacity to concerns about the risks of advanced AI models.
“I’m very happy to see any effort in openness,” Farhadi says. “I do believe a significant portion of the market will move towards open models. We need more of this.”
The enterprise angle behind DBRX
Databricks has a commercial reason to make DBRX open. Ali Ghodsi, CEO of Databricks, says many large companies outside the biggest tech firms have not widely used AI on their own data. The opportunity Databricks sees is in finance, medicine, and other industries where companies want ChatGPT-like tools but are cautious about sending sensitive information into the cloud.
Ghodsi calls the approach data intelligence, meaning intelligence that can understand a company’s own data. Databricks plans to customize DBRX for customers or build bespoke models from scratch for specific businesses.
For major companies, Ghodsi argues that the cost of building something at DBRX’s scale can make sense. He described that as the company’s big business opportunity.
The DBRX project also reflects Databricks’ acquisition strategy. In July last year, Databricks acquired MosaicML, a startup focused on building AI models more efficiently. That deal brought in several people involved in DBRX, including Frankle. The source notes that no one at either company had previously built something on that scale.
What made the model difficult to build
DBRX is a large language model based on an artificial neural network, a mathematical framework loosely inspired by biological neurons. Like other modern language models, it is generally based on the transformer, a type of neural network invented by a team at Google in 2017.
After the transformer emerged, OpenAI researchers trained versions of that model style on larger and larger collections of text from the web and other sources. The source says that process can take months, and that scaling the model and training data made outputs more capable, coherent, and seemingly intelligent.
Scale still drives much of the field. The source notes that OpenAI CEO Sam Altman has sought $7 trillion in funding for developing AI-specialized chips, according to The Wall Street Journal. But DBRX also shows that model design and training choices matter, not just raw size.
Frankle says dozens of decisions go into building an advanced neural network. Some knowledge comes from research papers, while other details circulate within the community. Keeping thousands of computers, network switches, and fiber-optic cables working together is one of the major technical challenges.
Data is another critical factor. Naveen Rao, a vice president at Databricks and previously founder and CEO of MosaicML, said data quality, cleaning, filtering, and preparation are central to model performance. That may explain why the training data is the one detail Databricks is not openly disclosing.
DBRX also uses an architecture known as mixture of experts. In that approach, only some parts of the model activate for a given query, depending on what the query contains. The source says this makes a model more efficient to train and operate.
The numbers show why that matters. DBRX has around 136 billion parameters, while Llama 2 has 70 billion, Mixtral has 45 billion, and Grok has 314 billion. But DBRX activates only about 36 billion parameters on average to process a typical query.
The result is a model Databricks presents as a new high point for open source LLMs. More broadly, DBRX suggests that the open source AI race is becoming less about whether open models can compete and more about how quickly they can close the gap with the most powerful closed systems.