# Unlocking Criminal Court Decisions: Flexible Information Extraction for Empirical Legal Research

DevFeed: [Unlocking Criminal Court Decisions: Flexible Information Extraction for Empirical Legal Research](<https://devfeed.tech/articles/unlocking-criminal-court-decisions-flexible-information-extraction-for-empirical-legal-research-115413.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.03051>)

Author: Tom\\'a\\v{s} Knap, Ivana Kvapil\\'ikov\\'a, Jan \\v{C}ern\\'y, Kl\\'ara Bendov\\'a, Jarom\\'ir \\v{S}avelka, Vojt\\v{e}ch Pour, Jakub Dr\\'apal

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [document ai](<https://devfeed.tech/topics/document-ai.md>), [human review](<https://devfeed.tech/topics/human-review.md>), [test](<https://devfeed.tech/topics/test.md>)

Tags: [criminal](<https://devfeed.tech/tags/criminal.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [legal](<https://devfeed.tech/tags/legal.md>), [legal-research](<https://devfeed.tech/tags/legal-research.md>), [research](<https://devfeed.tech/tags/research.md>)

## AI overview

The paper presents an LLM pipeline for structuring criminal verdicts through attribute exploration, text-span extraction, and value normalization, with annotation instructions that legal experts can refine. In DUI cases, revising instructions was associated with extraction accuracy rising from 52% to 70% and greater agreement between human annotators. Repeated runs of the same model agreed more with one another than with gold annotations, so consistency did not guarantee correctness; disagreements among annotators and models also exposed unclear instructions. For driving despite a ban, instructions specifying fixed categories and output formats yielded high agreement for most attributes.

## Source excerpt

arXiv:2611.03051v1 Announce Type: cross Abstract: Criminal verdicts contain descriptions of criminal behavior that can support quantitative legal research, but structuring them requires attributes and extraction rules that cannot always be fully specified in advance. We present an LLM-based pipeline that builds a structured dataset through attribute exploration, text span extraction, and value normalization, guided by annotation instructions that legal experts can refine. We evaluate it on two traffic offenses: driving under the influence (DUI) and driving despite a ban (DDB). For DUI, revising the instructions was associated with higher extraction accuracy (from 52% to 70% of correctly extracted values) and higher agreement between human annotators. Repeated runs of the same model agreed with each other more than with the gold annotations, showing that consistent output is not necessarily correct, while disagreement between annotators and between models helped reveal unclear instructions. For DDB, where the instructions defined fixed categories and output formats, agreement was high for most attributes.