# Can we trust LLMs to judge whether a research idea is novel?

DevFeed: [Can we trust LLMs to judge whether a research idea is novel?](<https://devfeed.tech/articles/can-we-trust-llms-to-judge-whether-a-research-idea-is-novel-102605.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/allenai/novelty-bench>)

Author: Kyle Wiggers

Published: 2026-10-27T16:12:42Z

Content type: news

Language: en

Sources: [Hugging Face](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [AI for science](<https://devfeed.tech/topics/ai-for-science.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [benchmark overfitting machine learning](<https://devfeed.tech/topics/benchmark-overfitting-machine-learning.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-systems](<https://devfeed.tech/tags/ai-systems.md>), [ai2](<https://devfeed.tech/tags/ai2.md>), [automated](<https://devfeed.tech/tags/automated.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [build](<https://devfeed.tech/tags/build.md>), [can](<https://devfeed.tech/tags/can.md>), [code](<https://devfeed.tech/tags/code.md>), [data](<https://devfeed.tech/tags/data.md>), [developing](<https://devfeed.tech/tags/developing.md>), [discovery](<https://devfeed.tech/tags/discovery.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [idea](<https://devfeed.tech/tags/idea.md>), [ideas](<https://devfeed.tech/tags/ideas.md>), [language](<https://devfeed.tech/tags/language.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [llms](<https://devfeed.tech/tags/llms.md>), [novel](<https://devfeed.tech/tags/novel.md>), [research](<https://devfeed.tech/tags/research.md>), [trust](<https://devfeed.tech/tags/trust.md>)

## AI overview

The study finds that LLM novelty judgments can change sharply with evaluation setup, especially for AI-generated ideas. Across six judges, a prompt change shifted GPT-5.4's accuracy on human-versus-generated pairs from 93.5% to 40.9%, and the judge reversed its verdict on more than half of the same pairs. Retrieval and added reasoning did not reliably improve results: a costly search agent scored below a simpler Claude Opus 4.6 setup, while two specialized evaluators were both less accurate and more expensive than a low-reasoning Opus 4.6 baseline with majority voting. The benchmark uses ICLR 2026 reviews to identify human ideas with stronger or weaker reviewer evidence of novelty, then pairs validated human ideas with generated ideas labeled as less novel by assumption. The authors caution that these generated-idea labels are weak and call for evaluation on the kinds of ideas judges will actually assess, with sensitivity to prompts and models measured.

## Source excerpt

Today we're releasing **Old Ideas, Novel Problems**, a new study of how reliably large language models can evaluate the novelty of scientific research ideas. Alongside the paper, we're publishing a benchmark, data, and code for testing novelty judges, plus an automated pipeline researchers can use to build new evaluation sets. At Ai2, we've been developing AI systems that support different parts of scientific discovery.