# Publications as benchmarks: auditing AI agents on spatial transcriptomics alignment

DevFeed: [Publications as benchmarks: auditing AI agents on spatial transcriptomics alignment](<https://devfeed.tech/articles/publications-as-benchmarks-auditing-ai-agents-on-spatial-transcriptomics-alignment-126180.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.11796>)

Author: Yiqun T. Chen, Jan Matthias, Jianing Yao, Hongyu Eric Zhao, Stephanie C. Hicks

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [auditing](<https://devfeed.tech/tags/auditing.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [publications](<https://devfeed.tech/tags/publications.md>), [q-bio-gn](<https://devfeed.tech/tags/q-bio-gn.md>), [spatial](<https://devfeed.tech/tags/spatial.md>), [stat-ap](<https://devfeed.tech/tags/stat-ap.md>)

## AI overview

The paper presents a framework for turning scientific publications into auditable AI-agent benchmarks and demonstrates it on spatial transcriptomics alignment. Across 121 tasks and 16 baseline methods, the strongest of six agents, GPT-6-Astra and Gemini-4-Argon, matched or exceeded every baseline on 80% of tasks, often with simple heuristics. Performance was robust to prompting and access to pre-installed packages. Building the benchmark also exposed errors in prior work and shortcuts in existing evaluation metrics. Agent solutions distilled into a reusable program ranked in the top 10% of baselines on unseen data. These findings are limited to the demonstrated tasks and evaluation setup.

## Source excerpt

arXiv:2611.11796v1 Announce Type: cross Abstract: AI agents are increasingly automating data analysis, but responsible adoption requires reliable performance evaluation, which is costly in specialized scientific domains. We introduce a framework that translates scientific publications into auditable benchmarks for AI agents and demonstrate its use on spatial transcriptomics alignment tasks. Across 121 tasks, 16 baseline methods, and six AI agents, the best agents (GPT-6-Astra and Gemini-4-Argon) matched or outperformed all baselines on 80% of tasks, often using surprisingly simple heuristics. Agent performance was robust to prompting and access to pre-installed packages. Benchmark construction also uncovered errors in prior work and shortcuts in current evaluation metrics. Finally, we show that agent-generated solutions can be distilled into a reusable program that ranks in the top 10% of baseline methods on unseen data. Our framework offers a scalable way to evaluate AI agents in specialized scientific domains by leveraging and stress-testing existing papers and metrics.