Building AI for scientific discovery takes more than a large model. It also requires computing power, carefully prepared research data, evaluation methods, and expertise across disciplines. The Trillion Parameter Consortium (TPC) is bringing researchers and institutions together to work on those challenges, with a particular focus on models with one trillion or more parameters.
A shared effort for scientific AI
The consortium brings together people from federal laboratories, research institutes, academia, and industry. Its goal is to create an open community where researchers who may already be working in small groups can coordinate on large-scale generative AI for scientific and engineering problems.
That coordination is intended to limit duplicated work and make it easier to share methods, tools, knowledge, and workflows. Rather than having each group solve the same practical problems separately, joint projects can help circulate approaches across the wider scientific and AI community.
The intended models would support more than questions within a single area of research. The consortium wants to develop systems that can answer domain-specific questions and synthesize knowledge across scientific disciplines. That ambition depends on both model development and the work of preparing scientific material for training.
What it takes to scale up
The TPC has identified several connected challenges: designing model architectures and training strategies that can scale, organizing and curating scientific data, adapting AI libraries for current and future exascale computing platforms, and building deep evaluation platforms.
Each of these areas affects whether a large model can be trained and assessed for scientific use. A model architecture must work at scale; its training data has to be organized; software libraries must make effective use of the available computing platforms; and evaluation must provide a way to examine how well the system performs.
The consortium has established working groups to address the complexities of building large-scale AI models. Those groups are expected to lead initiatives that use emerging exascale platforms to train large language models or alternative architectures for scientific research.
Computing resources and long training runs
Training models at this scale requires substantial computing resources. The article describes models with trillions of parameters as the limit of today's AI models and says only the largest commercial AI systems, such as GPT-4, currently reach that scale.
Resources for the planned training will come from several US Department of Energy (DOE) national laboratories and founding partners in Japan, Europe, and other countries. Even with those resources, training is expected to take several months.
This makes shared access to computing, data, and expertise central to the effort. The consortium is not only connecting research groups; it is also seeking to build a global network around the resources needed to pursue large scientific models.
Preparing data and connecting disciplines
Rick Stevens, associate laboratory director of computing, environment and life sciences at DOE’s Argonne National Laboratory and professor of computer science at the University of Chicago, described teams as beginning to develop frontier AI models for scientific use and prepare large collections of previously untapped scientific data for training.
That work points to a practical part of the consortium's mission: scientific AI depends on making research material usable alongside the infrastructure for training. Stevens said the TPC was created to accelerate these initiatives and develop the knowledge and tools needed for models that can connect ideas across fields.
The consortium's approach combines collaboration on projects with work on shared foundations: scalable methods, curated data, computing software, and evaluation. Whether these efforts produce models useful across disciplines will depend on how those pieces come together. For now, the TPC is organizing a broad research community around that challenge.