Google's Bard chatbot has received a new Gemini model, and early user evaluations suggest a meaningful step forward. In Chatbot Arena, the model has been rated near GPT-4 systems, giving Google a visible result for Bard before Gemini Ultra is released.
What Google Changed In Bard
Google's head of AI, Jeff Dean, announced the new Gemini model on X. According to the source article, the model belongs to the Gemini Pro family and carries the suffix "scale".
Dean said the Gemini updates have made Bard "much better" and given it "many more capabilities" compared with the launch in March. The source does not specify the exact technical changes behind the model.
The word "scale" is not explained by Dean in the article. The source notes that the name suggests it could be a larger, scaled version of the earlier Pro model, but that remains an interpretation rather than a confirmed specification.
Why The Chatbot Arena Result Matters
The new Bard model immediately took second place in the neutral Chatbot arena benchmark. It ranked ahead of the two GPT-4 models 0314 (March 2023) and 0613 (Summer 2023), while remaining behind GPT-4 Turbo (November 2023).
That placement is important because the source describes the new Bard model as the first to break into the GPT-4 phalanx. In plain terms, users comparing anonymous chatbot responses rated the new Bard experience highly enough to put it among leading GPT-4 systems on that leaderboard.
Chatbot Arena uses the Elo rating system, a method also used in chess and e-sports, to compare language models. Models face one another in anonymous, randomly selected duels, and users vote for the responses they prefer.
The platform collects user interactions, but only counts votes when the model names are unknown. If the user asks for the name of the model, that vote is not counted for the ranking.
Human Evaluation Versus Benchmarks
The source article emphasizes that Chatbot Arena measures user preference or perceived quality. That means its results can differ from synthetic benchmark results, where models are measured against fixed tasks or test sets.
Oriol Vinyals, head of deep learning at Google and co-lead of Gemini, pointed out that evaluating language models is "hard and nuanced," with academic evaluations leaking into the training datasets of AI models.
Vinyals called human evaluation "far superior," and said it "feels good that Bard Gemini Pro (free tier) climbed quite high on lmsys," suggesting that Gemini Ultra may perform even better.
This distinction matters for how the result should be read. A strong Chatbot Arena position says users preferred the model's answers in anonymous comparisons. It does not, by itself, prove that the model is stronger across every technical measure.
The Caveats Around The Ranking
The source article also gives an important reason for caution: the new Bard model has only been rated about 3,000 times so far, while the GPT-4 models have been rated up to 30,000 times.
That gap means the leaderboard position could still change as more users test the model. The source also notes that the gaps in the benchmark are generally small.
Another limitation is that alternative benchmark results for GPT Pro-Scale are not available. Without those results, the Chatbot Arena position is the main evidence presented in the source for how the new Bard model compares.
- Known from the source: Bard is using a new Gemini Pro family model with the suffix "scale".
- Known from the source: It ranked second in Chatbot Arena at the time described.
- Known from the source: It placed ahead of GPT-4 models 0314 (March 2023) and 0613 (Summer 2023), but behind GPT-4 Turbo (November 2023).
- Still uncertain: What "scale" means technically.
- Still uncertain: Whether the ranking will hold as more ratings come in.
What It Means For Gemini Ultra
Google's model lineup described in the source places Gemini Pro below Gemini Ultra. Ultra is described as Google's top-of-the-line and most capable AI model, but it had not yet been released in the article.
The Bard result therefore raises expectations for Gemini Ultra. The source says Gemini Ultra is expected to outperform Gemini Pro-Scale, and Vinyals's statement also suggests that Ultra may perform even better.
For now, the most careful conclusion is narrow but significant: Bard's new Gemini Pro "scale" model has produced a strong early showing in a human-evaluation setting. The result is promising for Google, but the evidence remains limited until more ratings and other benchmark results are available.