# Can we trust LLMs to judge whether a research idea is novel?

DevFeed: [Can we trust LLMs to judge whether a research idea is novel?](<https://devfeed.tech/articles/can-we-trust-llms-to-judge-whether-a-research-idea-is-novel-103563.md>)

Original publisher: [Read original article](<https://allenai.org/blog/novelty-bench>)

Published: 2026-10-27T08:00:00Z

Content type: release

Language: en

Sources: [Ai2 Blog](<https://devfeed.tech/sources/ai2-blog.md>)

Topics: [Parallel reasoning](<https://devfeed.tech/topics/parallel-reasoning.md>), [benchmark overfitting machine learning](<https://devfeed.tech/topics/benchmark-overfitting-machine-learning.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [about](<https://devfeed.tech/tags/about.md>), [absolute](<https://devfeed.tech/tags/absolute.md>), [acceptance](<https://devfeed.tech/tags/acceptance.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-generated-research](<https://devfeed.tech/tags/ai-generated-research.md>), [ai-systems](<https://devfeed.tech/tags/ai-systems.md>), [can](<https://devfeed.tech/tags/can.md>), [changes](<https://devfeed.tech/tags/changes.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [idea](<https://devfeed.tech/tags/idea.md>), [ideas](<https://devfeed.tech/tags/ideas.md>), [llms](<https://devfeed.tech/tags/llms.md>), [novel](<https://devfeed.tech/tags/novel.md>), [problems](<https://devfeed.tech/tags/problems.md>), [research](<https://devfeed.tech/tags/research.md>), [setup](<https://devfeed.tech/tags/setup.md>), [small](<https://devfeed.tech/tags/small.md>), [tests](<https://devfeed.tech/tags/tests.md>)

## AI overview

The study finds that LLM novelty judgments are highly sensitive to evaluation setup, especially for AI-generated research ideas. Across six judges, changing only a few prompt words shifted GPT-5.4 accuracy on human-generated idea pairs from 93.5% to 40.9%, reversing verdicts on over half the same pairs. Prompt effects varied by model, and retrieval or extra reasoning did not reliably improve performance. On the tested Human+Generated benchmark, a Claude Opus 4.6 prompt with low reasoning and three-call majority voting reached 0.854 macro-F1 for $2.59, outperforming the tested Idea Novelty Checker (0.642; $30.66) and AI Scientist evaluator (0.772; $82.05). An extra-high-reasoning GPT-5.6 Sol agent using iterative arXiv search scored 0.78 macro-F1 and cost roughly ten times more per decision than simpler setups. The benchmark uses ICLR 2026 peer-review signals for human ideas; AI-generated ideas carry weak labels based on an assumption, so the authors also examine instability independent of label correctness. They present the benchmark and data pipeline as starting tools, while concluding that reliable novelty evaluation remains unresolved.

## Source excerpt

Old Ideas, Novel Problems tests how reliably LLMs judge the novelty of research ideas, finding that small changes to the evaluation setup can produce very different verdicts.