# Double Blind Evals: Resolving the Dual Confidentiality Dilemma in AI Safety Auditing

DevFeed: [Double Blind Evals: Resolving the Dual Confidentiality Dilemma in AI Safety Auditing](<https://devfeed.tech/articles/double-blind-evals-resolving-the-dual-confidentiality-dilemma-in-ai-safety-auditing-125569.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.12604>)

Author: Andrew Trask, Sol Messing, Vai Pahwa, Pegah Maham, Rene Kolga, Austin Frantz, Andrew Tash, Kate Thomas, Sean McGregor, George Balston, Patricia Paskov, Miles Brundage, Akriti Vij, Bennett Hillenbrand,

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [ai security](<https://devfeed.tech/topics/ai-security.md>), [Cryptography](<https://devfeed.tech/topics/cryptography.md>), [Responsibility & Safety](<https://devfeed.tech/topics/responsibility-safety.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-safety](<https://devfeed.tech/tags/ai-safety.md>), [auditing](<https://devfeed.tech/tags/auditing.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cryptographic](<https://devfeed.tech/tags/cryptographic.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [data-leakage](<https://devfeed.tech/tags/data-leakage.md>), [deepmind](<https://devfeed.tech/tags/deepmind.md>), [evals](<https://devfeed.tech/tags/evals.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [google](<https://devfeed.tech/tags/google.md>), [google-cloud](<https://devfeed.tech/tags/google-cloud.md>), [google-cloud-platform](<https://devfeed.tech/tags/google-cloud-platform.md>), [google-deepmind](<https://devfeed.tech/tags/google-deepmind.md>), [h100](<https://devfeed.tech/tags/h100.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [inference](<https://devfeed.tech/tags/inference.md>), [ip](<https://devfeed.tech/tags/ip.md>), [model](<https://devfeed.tech/tags/model.md>), [safety](<https://devfeed.tech/tags/safety.md>)

## AI overview

The paper presents "double-blind evals" to protect both an evaluator's prompts and a model owner's proprietary weights and inference code. It uses secure enclaves, which encrypt data in RAM at the hardware level, and says the approach provides cryptographic guarantees that prompts are not shared with the model owner. In a pilot, the authors evaluated Google DeepMind's Gemini 2.5 Flash Lite on a selection of the private AILuminate benchmark using an NVIDIA H100 secure enclave on Google Cloud Platform and OpenMined's PySyft. The authors describe this as the first use of the technology for mutual secrecy between a proprietary frontier-class model and a closed benchmark.

## Source excerpt

arXiv:2611.12604v1 Announce Type: cross Abstract: AI safety evaluations often involve sensitive prompts and proprietary models. Evaluators want to keep their evaluation prompts secret from model owners to prevent purposeful or incidental data leakage, ensuring longer usefulness of their benchmark. Model owners in turn wish to keep their proprietary model weights and inference code within their own secure infrastructure to protect their IP. We present a way forward for "double-blind evals", which provide cryptographic guarantees that an evaluator's prompts are not shared with the model owner, while simultaneously maintaining the confidentiality of the model weights and inference code. To accomplish this, we use secure enclaves that encrypt data in RAM at the hardware level. In a world-first pilot, we test this approach by evaluating a proprietary Google DeepMind model (Gemini 2.5 Flash Lite) against a selection of the private AILuminate benchmark corpus, using an NVIDIA H100 secure enclave on Google Cloud Platform, and using OpenMined's PySyft to maintain data and model privacy. We believe this effort marks the first use of this technology to provide mutual secrecy for one organization's frontier-class proprietary model weights and another organization's closed benchmark.