AI benchmarks are only useful when the test is clean. Google Deepmind is now testing a way to protect that basic condition by keeping benchmark questions hidden from AI models and model providers alike.
The company says it is launching the first double-blind evaluation of a proprietary frontier AI model. The pilot involves the Singapore AI Safety Institute and other partners, and it tests a model from the Gemini Flash Lite line against confidential benchmarks.
Why AI benchmarks can lose their meaning
Benchmark contamination is the central problem. If a model has already encountered test questions during training, its score becomes harder to interpret. A high result may still look impressive, but it no longer proves the model can generalize to unseen tasks.
That matters because benchmarks are often used as shorthand for model progress. Developers, researchers, evaluators and institutions look at scores to understand where systems are strong, where they are weak and whether they can be trusted in demanding settings.
Google Deepmind frames the issue with a simple comparison: a test-taker who knows the questions in advance can produce a perfect score that does not mean very much. The same logic applies to AI models when benchmark prompts have leaked into training data or become available during optimization.
The problem is not only about public leaderboards. It also affects confidential evaluations, where the questions may be sensitive and the model under review may be proprietary. In those cases, both sides have something important to protect.
The tradeoff Google wants to remove
Before this approach, highly sensitive external evaluations often came with an uncomfortable choice, according to Deepmind. Evaluators could give test prompts to the model provider, which created a risk that the provider would see the questions. Or the model provider could share model weights with the evaluator, which created intellectual property risk.
That tradeoff can slow down or limit serious testing. A source of evaluation data may not want to expose confidential prompts. A model provider may not want to hand over the inner assets of a proprietary system. The result is a trust gap between the people who want to test the model and the people who control it.
The source article points to one recent example: the delayed evaluation for the ARC-AGI benchmark of Anthropic's Fable 5, tied to the AI company's 30-day data retention policy for its strongest models. The example shows how evaluation logistics can become difficult when confidentiality requirements and model access rules collide.
Google Deepmind's proposed answer is a double-blind setup. In that arrangement, the evaluator does not see the Gemini weights, and Google does not see the test prompts. The evaluation can proceed without either side giving up the asset it most needs to protect.
How the cryptographic setup works
The pilot uses Confidential Space from Google Cloud's confidential computing portfolio. The source describes the test data as being kept inside a cryptographic "box," so the model cannot later use those prompts to optimize itself specifically for the test.
The important feature is verification. The setup cryptographically verifies that the external benchmark data and the model remain private to their respective owners. That turns confidentiality from a promise into part of the technical process.
Until now, the source says external prompts were protected through zero-logging protocols and contractual safeguards. Those protections can matter, but they still depend heavily on rules, processes and trust between parties. Google Deepmind argues that adding technical and cryptographic protection is a significant step for secure model evaluation.
In practical terms, the double-blind model is meant to support three goals at once:
- Cleaner AI benchmarks: Test prompts stay hidden, reducing the risk of benchmark contamination.
- Protected model assets: Evaluators can run tests without seeing Gemini weights.
- Stronger evaluation trust: Both sides can verify privacy without relying only on contracts or logging policies.
Why sensitive evaluations are the key test
The method is especially relevant where the stakes of disclosure are high. Deepmind says the approach matters most for sensitive evaluations, including cybersecurity and tests run by government agencies.
Those areas can involve prompts or evaluation methods that organizations do not want to expose. At the same time, advanced AI models may need rigorous testing by independent organizations before they are trusted in sensitive contexts. A double-blind evaluation could make that kind of testing easier without forcing either party to surrender control.
The source also connects this work to data sovereignty and security. Independent organizations could test advanced models more rigorously while keeping their data private. Model providers could take part without releasing proprietary model weights.
That combination is the core appeal. The evaluator keeps the benchmark confidential. The model provider keeps the model protected. The public or institutional user gets a result that may be easier to trust because the test was designed to prevent the model from seeing the questions in advance.
A possible standard for model oversight
Google hopes the pilot will help set a new standard for model oversight. The broader aim is to make AI systems more reliable and more widely trusted by improving how they are evaluated.
That ambition depends on whether the method can work beyond a single pilot. The current test involves a Gemini Flash Lite line model, the Singapore AI Safety Institute and other partners. Google also says details on methodology and results are laid out in a technical report.
Still, the direction is clear from the source: AI benchmarks need stronger protections if they are going to remain credible. As models become more capable and evaluations become more sensitive, the process for testing them has to protect both the benchmark and the model.
For now, the significance of Google's effort is not that it ends the debate over AI benchmark trust. It is that it targets one of the debate's most basic weaknesses: a benchmark cannot measure unseen performance if the questions are no longer unseen.