Google DeepMind tested a model with neither side seeing the other's data

Google DeepMind published what it calls the first double-blind evaluation of a proprietary model close to the most capable ones. Amanda Caswell reported on it in The New Stack on 27 August.

The arrangement: Gemini's weights stayed hidden from the evaluators, and the test questions stayed hidden from Google. The pilot ran Gemini 2.5 Flash Lite against private benchmarks from MLCommons and the Singapore AI Safety Institute. Google is not promoting a score. The method is the point.

The problem it addresses

Google's technical report cites earlier research that found signs of benchmark leakage in about half of the 31 models it tested, and another study this year finding contamination inflates scores, particularly for larger models.

Keeping benchmarks private seems like the obvious fix, and with a closed model it does not work. Evaluators have to send their questions through the provider's API, at which point the provider has seen the test. The alternative is the provider handing over its weights, which no leading AI lab is going to do.

How the enclave works

Google Cloud Confidential Space, an Nvidia H100 Confidential GPU, and Intel TDX host memory encryption.

DeepMind supplies weights and inference code. The evaluator supplies benchmark prompts and evaluation code. Both travel over encrypted connections into the enclave, where the evaluation runs with neither party reaching the other's assets. Weights are stored in hardware-encrypted GPU memory, prompts in encrypted host memory. Afterwards the evaluator receives the permitted results and the environment is destroyed.

Before either side sends anything, remote attestation confirms the enclave is running the software both agreed to.

Encryption alone is not enough, because code inside the enclave could still send data out. OpenMined's PySyft handles that: both parties approve the code in advance, and sensitive parts of the evaluation are blocked from making external connections.

What it does not fix

The report is candid, and the caveats are the most useful part.

Compute overhead is under 5 percent. The real obstacles are legal agreements and cross-organisation code reviews.

Trust is reduced rather than removed, and part of it now rests on the hardware. The arrangement assumes the cloud provider and the chipmaker are not cooperating to defeat it. Google also still has a role in verification: the Confidential Space guest operating system is open source and its build process has been externally validated, but individual builds use private signing keys and cannot be independently reproduced, and Google's own services sign and verify the attestation report.

MLCommons adds that the benchmark itself still needs careful management however well the questions are protected.

The pilot used a single H100 80GB. Clusters of H100 and B200 GPUs over encrypted links are being looked at for models too large to fit on one.

Every score you already rely on

Most teams will never run one of these. The reason to read it is what it says about every score you currently rely on.

A benchmark number from a lab that could see the test is a claim, not a measurement. That has always been true and there was no practical alternative. Now there is one, which means "was this evaluated blind" becomes a question you can reasonably ask.

Source

• Google found a way to test Gemini without seeing the questions, The New Stack, 27 August 2026