# Can LLMs Reason in a Legally Meaningful Manner? A Small-scale Study on European Court of Human Rights Cases

DevFeed: [Can LLMs Reason in a Legally Meaningful Manner? A Small-scale Study on European Court of Human Rights Cases](<https://devfeed.tech/articles/can-llms-reason-in-a-legally-meaningful-manner-a-small-scale-study-on-european-court-of-human-rights-cases-120074.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2608.17168>)

Author: Amogh Raina, Ilias Chalkidis, Daniel Hershcovich, Henrik Palmer Olsen

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [human review](<https://devfeed.tech/topics/human-review.md>), [AI response quality](<https://devfeed.tech/topics/ai-response-quality.md>)

Tags: [court](<https://devfeed.tech/tags/court.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [forecasting](<https://devfeed.tech/tags/forecasting.md>), [gpt](<https://devfeed.tech/tags/gpt.md>), [gpt-5-4](<https://devfeed.tech/tags/gpt-5-4.md>), [human-rights](<https://devfeed.tech/tags/human-rights.md>), [legal](<https://devfeed.tech/tags/legal.md>), [llm-as-a-judge](<https://devfeed.tech/tags/llm-as-a-judge.md>), [llm-evaluation](<https://devfeed.tech/tags/llm-evaluation.md>), [llms](<https://devfeed.tech/tags/llms.md>), [openai](<https://devfeed.tech/tags/openai.md>), [predictions](<https://devfeed.tech/tags/predictions.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [prompting](<https://devfeed.tech/tags/prompting.md>), [proxy](<https://devfeed.tech/tags/proxy.md>), [quality](<https://devfeed.tech/tags/quality.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [tasks](<https://devfeed.tech/tags/tasks.md>)

## AI overview

In this small-scale study of European Court of Human Rights case forecasting, OpenAI GPT 5.4 produced analyses that were structurally complete but substantively shallow, with legal reasoning quality judged far from ideal. An expert-curated prompt made reasoning more comprehensive but did not improve prediction accuracy over the other prompting settings. LLM evaluators were internally consistent but aligned only weakly with trained human annotators, so the authors caution against treating automated evaluation as a substitute for human assessment or prediction accuracy as a proxy for reasoning quality.

## Source excerpt

arXiv:2608.17168v3 Announce Type: replace-cross Abstract: Reasoning has become a standard technique and feature for contemporary LLMs; however, its application and quality in the context of demanding legal-oriented tasks, such as legal case forecasting, remain under explored. We investigate how LLMs reason in the context of legal case forecasting, using legal cases from the European Court of Human Rights (ECtHR) as a testbed. We evaluate OpenAI GPT 5.4, a recent top-tier LLM, by exploring alternative prompting strategies that are more or less suggestive of what counts as legally meaningful reasoning in the context of ECtHR jurisprudence. We present our findings derived from assessing the model's responses with both human and LLM evaluation. We find that the examined model scores far from ideal in legal reasoning, the model produces structurally complete but substantively shallow analyses, and that LLM-as-a-Judge evaluators are internally consistent yet align only weakly with our trained annotators, i.e., reliable but not a valid substitute for human evaluation. Overall, the expert-curated prompt leads to more comprehensive reasoning, which does not result in more accurate predictions compared to the other examined settings. Based on our findings, we urge the community not to rely solely on automated LLM-based evaluation and to avoid using task accuracy as an appropriate proxy for reasoning quality.