Inflection presents its new language model, Inflection-2, as a major step up from its predecessor. The company says it beats Google PaLM 2 Large on most standard benchmarks and Claude 2 when chain-of-thought reasoning is used, while still falling short of GPT-4 in some areas.
Benchmark gains come with important caveats
Inflection says Inflection-2 improves factual knowledge, style control and reasoning compared with Inflection-1, which was released in July and was roughly on par with GPT-3.5 and PaLM-540B. The company describes its new model as second only to GPT-4, a claim that reflects its own assessment of the comparisons it reports.
The model was trained on 5,000 NVIDIA H100 GPUs, using a mixing accuracy of fp8, for about 10²⁵ FLOPs. Inflection says this places the training effort in the same class as Google's flagship PaLM 2 Large, which will soon be replaced by Gemini.
On most of the standard tests cited by the company, Inflection-2 scores above PaLM 2 Large. Those include MMLU, a benchmark covering language-related tasks from high school to professional level, as well as TriviaQA, HellaSwag and GSM8k.
On HellaSwag 10-shot, Inflection-2 scored 89.0, compared with GPT-4's 95.3. That result puts the models closer on this particular test, but does not establish that Inflection-2 matches GPT-4 overall. Inflection also says its model outperforms Claude 2 when both are assessed with chain-of-thought reasoning, an optimized prompting process.
Coding and math remain weaker areas
The company acknowledges that Inflection-2 falls well short of GPT-4 on coding and math tasks. It says the model was not optimized for coding, and notes that it performs better than Meta's Llama 2 in those areas.
That distinction matters when interpreting a broad claim about model performance. Scores on language benchmarks can show strengths on particular tasks, while the reported gaps in coding and math point to limits users may encounter with practical problem solving. Inflection's results describe a model with uneven capabilities rather than one that leads every comparison.
Pi chatbot and a larger model are next
Inflection plans to move its Pi chatbot to Inflection-2. The company is upgrading the infrastructure supporting the chatbot from Nvidia A100 to H100 GPUs, which should speed up inference, or the processing of a user's input by the model.
Inflection says that despite its multiple size of 175 billion parameters, Inflection-2 should be cheaper and faster than Inflection-1. The company is also planning to train larger models using the full capacity of its 22,000-GPU cluster. It says the next model will be about ten times larger and released in about six months.
The company has also voluntarily signed on to the White House's July 2023 commitments on safety and responsibility. Its stated plans combine a near-term update to Pi with longer-term work on larger models, though the source does not describe additional details about how those commitments will be applied to the products.
Inflection's company and backing
Inflection went public in March 2022. Its founders are LinkedIn founder Reid Hoffman, Deepmind co-founder Mustafa Suleyman, and former Deepmind researcher Karén Simonyan. The startup focuses on using natural language as a personal interface to computers.
Inflection AI closed a $225 million investment round in May 2022. In June 2023, it announced a further investment round in which Microsoft, Reid Hoffman, Bill Gates, Eric Schmid, and Nvidia invested a total of $1.3 billion. At the time, the company was valued at $4 billion.
The company has also added researchers and product leaders, including Heinrich Kuttler of Meta AI, Maarten Bosma and Rewon Child, formerly of Google Brain, and former Deepmind and Google product manager Joe Fenton. The people and funding behind Inflection help explain the company's capacity to build and deploy models, while its benchmark claims remain the basis for judging Inflection-2's reported performance.