FlexOlmo offers a practical answer to a familiar AI problem: many organizations have useful data for training language models, but they cannot simply send that data outside their own systems. Developed by the Allen Institute for AI, the method is designed for collaborative training where the data stays local and the group shares model weights instead.
Why local data is hard to use together
In regulated industries, valuable datasets often sit behind strict access controls. That can limit how organizations contribute to large language model training, even when their data could improve performance in important domains.
FlexOlmo addresses that constraint with a mixture-of-experts setup. Each expert is trained independently on a closed dataset. Rather than pooling raw data, each participating organization trains its own expert locally and contributes the resulting model weights to the shared system.
This matters because the collaboration is built around what can be shared more safely. The source data does not need to leave the organization that controls it. The shared artifact is the trained expert, not the dataset itself.
How FlexOlmo keeps experts aligned
The central technical challenge is not just privacy. It is coordination. If different experts are trained separately, they need a common point of reference so they can be combined later into one working model.
FlexOlmo uses a frozen public model for that role. The public expert stays fixed during training, while new experts learn from local data. Because each new expert is trained against the same reference, the experts can later be combined without additional retraining.
That fixed reference is what turns separate local training runs into a coordinated system. It gives the model a way to absorb specialized information from different sources while avoiding the need to centralize the data itself.
Data sources can be switched on or off
FlexOlmo is also built for cases where access to data may need to change over time. Data sources can be activated or deactivated depending on how the model will be used.
The source article gives a simple example: toxic content might be useful for research, but excluded from general use. In that kind of setting, the value of a dataset depends on the application, not only on whether it improves a benchmark.
The researchers tested this flexibility by removing the news expert. News-related performance dropped, as expected, while results in other areas stayed stable. That result is important because it shows that removing one source does not necessarily disrupt the rest of the system.
This design also helps when licenses change or usage rights expire. A data source can be deactivated later without retraining the entire model. The final model has 37 billion parameters, with 20 billion active.
What the tests showed
The team evaluated FlexOlmo with public data and seven specialized datasets: News, Creative Writing, Code, Academic Papers, Educational Text, Math, and Reddit content.
Across 31 tasks, FlexOlmo delivered an average improvement of 41 percent over a model trained only on public data. In general benchmarks, it also outperformed a hypothetical model that had access to all data using the same computational effort.
Only a model trained on the entire dataset with double the resources performed slightly better. That comparison matters because FlexOlmo is not presented only as a privacy workaround. In the reported tests, it also produced strong performance under realistic constraints.
The results point to a useful tradeoff. Organizations can keep training data under local control while still contributing to a broader LLM. At the same time, the model can benefit from specialized sources such as code, math, academic papers, or educational text.
Privacy tools and traceability
Because participants share trained model weights rather than raw datasets, the reported risk of data leakage is minimal. In testing, attacks to recover training data succeeded just 0.7 percent of the time.
For organizations with especially sensitive data, FlexOlmo also supports differentially private training. Each participant can enable that option independently, which means one organization can apply stronger privacy protection without forcing the same choice on every other participant.
The Allen Institute has also released OLMoTrace, a tool for tracing language model outputs back to their training sources. In a system where different data owners contribute different experts, that kind of tracing can help connect model behavior to the material that shaped it.
The broader idea behind FlexOlmo is straightforward: collaboration does not have to mean handing over the data itself. By separating local expert training from shared model assembly, the system gives organizations a way to participate in LLM development while keeping tighter control over sensitive, licensed, or restricted sources.