# CulturalCrosswords: A Multilingual Benchmark comprised of Answer Constraints and Grid Completion

DevFeed: [CulturalCrosswords: A Multilingual Benchmark comprised of Answer Constraints and Grid Completion](<https://devfeed.tech/articles/culturalcrosswords-a-multilingual-benchmark-comprised-of-answer-constraints-and-grid-completion-125872.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.10044>)

Author: Isaac Saxton-Knight, Gabriel Saxton-Knight, Diego Oliveira, Tidor-Vlad Pricope, Aston Garratt, Jocelyn D'Arcy

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Localization (l10n)](<https://devfeed.tech/topics/localization.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>)

Tags: [alternatives](<https://devfeed.tech/tags/alternatives.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [completion](<https://devfeed.tech/tags/completion.md>), [crosswords](<https://devfeed.tech/tags/crosswords.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [grid](<https://devfeed.tech/tags/grid.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [languages](<https://devfeed.tech/tags/languages.md>)

## AI overview

CulturalCrosswords evaluates how answer constraints and grid completion affect LLM performance on culturally specific riddles. The benchmark contains 53 crosswords built from crowdsourced riddles in 42 languages and tests 11 model configurations across five formats, from plain clue batches to completed grids. For 962 clues with alternative answers, adding answer lengths raises agreement with the reference by 22.8 percentage points; crediting accepted alternatives yields an 8.4-point increase, suggesting some benefit comes from steering models toward a required answer form. Crossword input raises the share of crossings with mutually compatible answers by 9.7 points, while completed-grid tasks expose substantial differences between models. The benchmark separates matching a reference answer from producing answers that fit together in the grid.

## Source excerpt

arXiv:2611.10044v1 Announce Type: cross Abstract: When several answers fit a riddle, exact-match scoring can penalise a valid response. CulturalCrosswords tests how answer constraints affect large language models (LLMs) solving cultural riddles, whose solutions draw on knowledge specific to a community. We constructed 53 crosswords from crowdsourced riddles in 42 languages and evaluated 11 LLM configurations on five versions of each clue set: plain batches, batches with lengths, batches with lengths and partial-answer hints, and crosswords requesting an answer list or a completed grid. On 962 clues with alternative answers, lengths improve agreement with the reference by 22.8 percentage points, compared with 8.4 points when accepted alternatives also receive credit. Part of the gain therefore appears to come from selecting a required answer form. Crossword input increases the share of all crossings with compatible answers by 9.7 points; requiring a completed grid reveals large differences between models. The benchmark distinguishes agreement with a reference from the ability to produce answers that fit together and occupy the correct cells.