# llm-reasoning

LLM reasoning is the use and development of large language models for complex, multi-step reasoning, including reinforcement-learning training, automated reasoning-data construction, and test-time scaling.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Best Agent Frameworks for Production Use

DevFeed: [Best Agent Frameworks for Production Use](<https://devfeed.tech/articles/best-agent-frameworks-for-production-use-146945.md>)

Original publisher: [Read original article](<https://www.swytchcode.com/content/best-agent-frameworks-production>)

Author: Swytchcode Team

Published: 2026-11-20T09:00:00Z

Content type: comparison

Language: en

Sources: [Swytchcode](<https://devfeed.tech/sources/swytchcode.md>)

Topics: [Microsoft Agent Framework](<https://devfeed.tech/topics/microsoft-agent-framework.md>), [Multi Agent Systems](<https://devfeed.tech/topics/multi-agent-systems.md>), [Agent Harness](<https://devfeed.tech/topics/agent-harness.md>), [ai-governance](<https://devfeed.tech/topics/ai-governance.md>), [audit](<https://devfeed.tech/topics/audit.md>), [agent observability](<https://devfeed.tech/topics/agent-observability.md>), [Production Engineering](<https://devfeed.tech/topics/production-engineering.md>), [SDK](<https://devfeed.tech/topics/sdk.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [agent-frameworks](<https://devfeed.tech/tags/agent-frameworks.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [audit](<https://devfeed.tech/tags/audit.md>), [best](<https://devfeed.tech/tags/best.md>), [permissions](<https://devfeed.tech/tags/permissions.md>), [production](<https://devfeed.tech/tags/production.md>), [retries](<https://devfeed.tech/tags/retries.md>)

### AI overview

The excerpt's central finding is that agent frameworks manage reasoning and tool orchestration, while API execution controls such as credentials, policy checks, idempotent retries, and independent audit generally require another layer. LangGraph is suited to branching, stateful workflows with checkpointing and human review; OpenAI Agents SDK offers native handoffs and scoped guardrails, with gaps around handoffs, hosted tools, retries, and audit; Claude Agent SDK's local tool approvals fit supervised coding and internal tools. CrewAI adds runtime controls and credential hygiene, while Vercel AI SDK offers provider and runtime flexibility but leaves credentials, sandboxing, approvals, and idempotency to the caller. Microsoft Agent Framework provides broad orchestration, with governance positioned in Azure AI Foundry. Temporal adds durable state and retries after failures, but does not decide whether a call is authorized. The excerpt describes a separate execution layer as handling allowlists, live schema validation, policy, credential injection, safe retries, response checks, and audit.

### Source excerpt

We rank six agent frameworks used in production as of November 2026 and show why none of them handle API permissions, retries, or audit by design.

## Your AI red team may be finding the same prompt injection thousands of times

DevFeed: [Your AI red team may be finding the same prompt injection thousands of times](<https://devfeed.tech/articles/your-ai-red-team-may-be-finding-the-same-prompt-injection-thousands-of-times-145725.md>)

Original publisher: [Read original article](<https://www.helpnetsecurity.com/2026/11/20/ai-prompt-injection-red-team/>)

Author: Mirko Zorz

Published: 2026-11-20T06:30:20Z

Content type: news

Language: en

Sources: [Help Net Security](<https://devfeed.tech/sources/help-net-security.md>)

Topics: [prompt injection](<https://devfeed.tech/topics/prompt-injection.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [qwen](<https://devfeed.tech/topics/qwen.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [ai-red-team](<https://devfeed.tech/tags/ai-red-team.md>), [attacks](<https://devfeed.tech/tags/attacks.md>), [automated](<https://devfeed.tech/tags/automated.md>), [cybersecurity](<https://devfeed.tech/tags/cybersecurity.md>), [defense](<https://devfeed.tech/tags/defense.md>), [don-t-miss](<https://devfeed.tech/tags/don-t-miss.md>), [features](<https://devfeed.tech/tags/features.md>), [hot-stuff](<https://devfeed.tech/tags/hot-stuff.md>), [llms](<https://devfeed.tech/tags/llms.md>), [news](<https://devfeed.tech/tags/news.md>), [prompt-injection](<https://devfeed.tech/tags/prompt-injection.md>), [prompt-injection-attacks](<https://devfeed.tech/tags/prompt-injection-attacks.md>), [red-team](<https://devfeed.tech/tags/red-team.md>), [red-teaming](<https://devfeed.tech/tags/red-teaming.md>), [research](<https://devfeed.tech/tags/research.md>), [team](<https://devfeed.tech/tags/team.md>), [workflow](<https://devfeed.tech/tags/workflow.md>)

### AI overview

In three of four setups, retraining on more successful attacks from automated red-team tools left agents easier to hijack. The researchers found that the tools often repeated a few techniques with altered wording, so attack counts overstated coverage. Their PIPETTE system rewards attacks using new techniques and trains agents to check whether each action serves the user's request. On AgentDojo, standard injection attacks succeeded in under 0.5% of attempts on average against PIPETTE-trained small models; adaptive attackers reached up to 10% on the 40 selected weak-spot tasks. The defense also preserved task completion: hardened Qwen3-4B completed slightly more clean tasks than the original, while PromptGuard blocked somewhat more under one attack but completed the user's task only 16.19% of the time, versus 58.10% for PIPETTE. Results are limited to open-weight models and small task sets; a larger attacker budget may do better, and the researchers have not verified their technique-coverage measure against human judgment or established that narrow coverage alone explains the gap.

### Source excerpt

AI agents retrained on a bigger pile of prompt injection attacks came out easier to hijack in three of four tests run by researchers at the University of Waterloo, Zhejiang University and KTH Royal Institute of Technology. The automated red-teaming tool that found more working attacks produced the weaker defense. Prompt injection purple-teaming workflow (Source: Research paper) Someone hides instructions in an email, a web page or a tool's output, and the AI agent that ... More -> The post Your AI red team may be finding the same prompt injection thousands of times appeared first on Help Net Security.

## Learning Which Evidence-Seeking Decisions Matter: Calibrated Counterfactual Replay for Language Agents

DevFeed: [Learning Which Evidence-Seeking Decisions Matter: Calibrated Counterfactual Replay for Language Agents](<https://devfeed.tech/articles/learning-which-evidence-seeking-decisions-matter-calibrated-counterfactual-replay-for-language-agents-144902.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2606.05263>)

Author: Renwei Meng, Chengyi Sun, Hongyi Zhang, Xiaoyi Hou

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Calibrated uncertainty](<https://devfeed.tech/topics/calibrated-uncertainty.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [learning](<https://devfeed.tech/tags/learning.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>)

### AI overview

PCCC-RL estimates which evidence-seeking actions matter by restoring a saved tool state, comparing the chosen action with a feasible alternative, and letting the same frozen policy complete both branches; the difference in final score estimates the action's value. To reduce replay costs, it predicts credit, checks a random subset, and corrects prediction errors using recorded selection probabilities. Alternatives depend only on the prefix, preserving a usable gradient baseline. On HotpotQA and MuSiQue, joint exact match--requiring both a correct answer and complete supporting evidence--was 56.71% and 36.82%, respectively, 4.19 and 3.86 points above a baseline with the same replay access. Gains concentrate in complete evidence chains and accumulate over many updates; single-update effects remain uncertain, while citation coverage and distinctions such as "none" versus "not all" remain open.

### Source excerpt

arXiv:2606.05263v2 Announce Type: replace Abstract: Language agents answer questions by searching, reading, and checking claims. A reward on the final answer says little about which of those decisions earned it, since an agent can reach the right string after redundant searches or with half the evidence missing. We treat the question as an experiment. At a saved decision point, PCCC-RL, for prefix-conditioned counterfactual credit, restores the tool state, runs the action the agent chose alongside a feasible alternative, and lets the same frozen policy finish both branches. The difference in final score then samples what the decision was worth. Replaying everything is too expensive, so the method predicts each action's credit up front, checks a random subset, and uses the recorded selection probabilities to correct the errors of that prediction, spending the budget where a further check can still move the training signal. Alternatives are drawn from the prefix alone, which keeps the comparison a usable gradient baseline. On HotpotQA and MuSiQue, joint exact match, requiring a correct answer together with complete supporting evidence, reaches 56.71% and 36.82%, gains of 4.19 and 3.86 points over a baseline with the same replay access. The gain concentrates in complete evidence chains and accumulates over many updates. Single-update effects stay uncertain, and citation coverage and scope distinctions such as "none" versus "not all" remain open.

## Keystone: Judging Previews at Key Steps for Test-Time Search in Diffusion Models

DevFeed: [Keystone: Judging Previews at Key Steps for Test-Time Search in Diffusion Models](<https://devfeed.tech/articles/keystone-judging-previews-at-key-steps-for-test-time-search-in-diffusion-models-145530.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25530>)

Author: Jiajun Fan, Chaoran Cheng, Ziwen Wang, Changyi Yang, Bangji Yang, Qinglong Zheng, Ge Liu

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [stable-diffusion](<https://devfeed.tech/topics/stable-diffusion.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Image](<https://devfeed.tech/topics/image.md>), [diffusion-transformers](<https://devfeed.tech/topics/diffusion-transformers.md>)

Tags: [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [diffusion-models](<https://devfeed.tech/tags/diffusion-models.md>), [keystone](<https://devfeed.tech/tags/keystone.md>), [preview](<https://devfeed.tech/tags/preview.md>), [search](<https://devfeed.tech/tags/search.md>), [test](<https://devfeed.tech/tags/test.md>), [text-to-image](<https://devfeed.tech/tags/text-to-image.md>)

### AI overview

Keystone is a training-free method for judging diffusion-model previews early in denoising, where the abstract says outcomes are often already determined by the seed or a sampler's initial stochastic steps. Instead of applying reward models designed for finished images--which read early previews near chance on several prompt types--Keystone turns prompts into typed questions about objects, counts, colours, and positions, then uses an off-the-shelf multimodal decision model to answer them. On GenEval with SD3.5-Medium, its adaptive search and pruning reportedly beat best-of-8 with every other verifier using less than half the generator compute. As a drop-in verifier, its score also improved best-of-N, progressive seed pruning, and FK steering there. Without retuning, both searches remained above the strongest other verifier's best-of-N on average across four base models and three benchmarks.

### Source excerpt

arXiv:2611.25530v1 Announce Type: new Abstract: Text-to-image diffusion and flow models yield a preview, their clean-image estimate, at every denoising step. When they miscount, drop or misplace objects, test-time search is in a bind: its verifiers are reward models built for finished images, so judging early misreads blurry previews, and judging late, as best-of-N does, pays to fully denoise doomed candidates. We find that the image is not the problem: the outcome is settled as denoising starts, by the seed or by a stochastic sampler's first steps, and already shows in early previews; what fails on them is the judge, as reward models read them near chance on several kinds of prompts. We introduce Keystone, a training-free judge of previews that compiles a prompt into typed questions about its objects, counts, colours and positions and has an off-the-shelf multimodal decision model answer them. Asked this way, the judge reads counts and positions from early previews better than reward models read them from finished images. Searching at these key steps on GenEval with SD3.5-Medium, Keystone's adaptive search and pruning significantly outperform best-of-8 with every other verifier at under half its generator compute, and as a drop-in verifier there, the Keystone score significantly improves best-of-N, progressive seed pruning and FK steering; without retuning, both searches stay significantly above the strongest other verifier's best-of-N on average over four base models and three benchmarks. Keystone thus moves test-time judgment to the key steps where the outcome already shows.

## Generalist Meets Specialists: An Evolving Multimodal Agent with Persistent Tool Orchestration for Traffic Scenario Understanding

DevFeed: [Generalist Meets Specialists: An Evolving Multimodal Agent with Persistent Tool Orchestration for Traffic Scenario Understanding](<https://devfeed.tech/articles/generalist-meets-specialists-an-evolving-multimodal-agent-with-persistent-tool-orchestration-for-traffic-scenario-understanding-145597.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25868>)

Author: Linjia Kang, Binhan Jiang, Juan Yang, Zhiqin Song, Xiangsen Wu, Qionglin Ren, Jiang Liu, Zhongjiang He, Hao Sun

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Multi-step AI processes](<https://devfeed.tech/topics/multi-step-ai-processes.md>), [agentic workflows](<https://devfeed.tech/topics/agentic-workflows.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [adoption](<https://devfeed.tech/tags/adoption.md>), [agent](<https://devfeed.tech/tags/agent.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [challenges](<https://devfeed.tech/tags/challenges.md>), [collaboration](<https://devfeed.tech/tags/collaboration.md>), [completion](<https://devfeed.tech/tags/completion.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cost-savings](<https://devfeed.tech/tags/cost-savings.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [error](<https://devfeed.tech/tags/error.md>), [evolution](<https://devfeed.tech/tags/evolution.md>), [execution](<https://devfeed.tech/tags/execution.md>), [fast](<https://devfeed.tech/tags/fast.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [generalist](<https://devfeed.tech/tags/generalist.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [orchestration](<https://devfeed.tech/tags/orchestration.md>), [specialists](<https://devfeed.tech/tags/specialists.md>), [tool](<https://devfeed.tech/tags/tool.md>), [traffic](<https://devfeed.tech/tags/traffic.md>)

### AI overview

StarTV-Agent combines policy learning with reusable tool workflows to balance adaptive reasoning with lower-cost handling of recurring traffic-understanding tasks. During training, Role-Sensitive Process Optimization combines process and terminal advantages, while Weakness-Driven Demonstration Learning uses validated teacher examples aimed at the student's weak questions. At inference, Lightweight Adaptive Workflow Reuse selects a suitable workflow and binds task-specific arguments and resource references; tasks without a suitable workflow use a slower ReAct-based path. Label-free feedback from execution completion, workflow adoption, and cost savings guides reuse and memory updates. Across three benchmarks, the abstract reports up to 8.0% relative performance improvement and up to 52.1% lower token consumption.

### Source excerpt

arXiv:2611.25868v1 Announce Type: new Abstract: Traffic scenario understanding requires combining fine-grained visual evidence with spatial and temporal reasoning. Tool-augmented vision-language models (VLMs) can use specialized perception tools, but effective collaboration faces two challenges: terminal rewards provide limited guidance for intermediate decisions and error recovery, while repeated planning adds unnecessary cost across similar tasks. Echoing the fast-slow dichotomy of human cognition, we propose StarTV-Agent, which combines policy learning with experience evolution to support adaptive reasoning when needed and efficient reuse of prior experience when possible. During training, Role-Sensitive Process Optimization (RSPO) combines process advantages with terminal advantages, while Weakness-Driven Demonstration Learning (WDL) provides validated teacher demonstrations targeting the student's weak questions. During inference, Lightweight Adaptive Workflow Reuse (LAWR) enables a fast path by selecting suitable workflows and binding task-specific arguments and resource references, while a slow ReAct-based path handles tasks without suitable workflows. We further design label-free feedback based on execution completion, workflow adoption, and cost savings to guide workflow reuse and memory updates. Across three challenging benchmarks, StarTV-Agent achieves up to 8.0% relative performance improvement, while reducing token consumption by up to 52.1%.

## The Price of an Explanation: What Demanding Verifiable Artifacts Costs, and What It Buys Back

DevFeed: [The Price of an Explanation: What Demanding Verifiable Artifacts Costs, and What It Buys Back](<https://devfeed.tech/articles/the-price-of-an-explanation-what-demanding-verifiable-artifacts-costs-and-what-it-buys-back-145115.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25896>)

Author: Edgar A. Du\\'e\\~nez-Guzm\\'an, Eduardo Due\\~nez, Edgar Said Hernandez Sanchez, Suzanne Sadedin

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [few-shot](<https://devfeed.tech/topics/few-shot.md>), [benchmark overfitting machine learning](<https://devfeed.tech/topics/benchmark-overfitting-machine-learning.md>), [reproducibility](<https://devfeed.tech/topics/reproducibility.md>)

Tags: [agi](<https://devfeed.tech/tags/agi.md>), [artifacts](<https://devfeed.tech/tags/artifacts.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [bottleneck](<https://devfeed.tech/tags/bottleneck.md>), [budget](<https://devfeed.tech/tags/budget.md>), [costs](<https://devfeed.tech/tags/costs.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-ne](<https://devfeed.tech/tags/cs-ne.md>), [decoder](<https://devfeed.tech/tags/decoder.md>), [error-correction](<https://devfeed.tech/tags/error-correction.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [llm](<https://devfeed.tech/tags/llm.md>), [mitigation](<https://devfeed.tech/tags/mitigation.md>), [program](<https://devfeed.tech/tags/program.md>), [quantum](<https://devfeed.tech/tags/quantum.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [score](<https://devfeed.tech/tags/score.md>), [search](<https://devfeed.tech/tags/search.md>), [synthesis](<https://devfeed.tech/tags/synthesis.md>), [tasks](<https://devfeed.tech/tags/tasks.md>), [test](<https://devfeed.tech/tags/test.md>), [verification](<https://devfeed.tech/tags/verification.md>)

### AI overview

On ARC-AGI-2, requiring an LLM to produce a verifiable program reduced performance from 93/120 tasks (77%) to 68/120 (57%). The authors report that most answer-only wins remained out of reach for the program-producing configurations they tested, even when a program matched every demonstration, pointing to generalization as the key bottleneck. A short evolutionary search over two sampled programs improved on best-of-two sampled code by 9.2 ± 1.4 tasks without additional model cost. Checking programs against training examples also enabled early stopping, reducing budget use by up to 38% for best-of-n sampling and 49% for the evolved arm; answer-only runs could not stop this way. The same methodology was applied to quantum error-correction decoder synthesis, producing interpretable decoders evaluated on held-out shots against a certified panel of published decoders.

### Source excerpt

arXiv:2611.25896v1 Announce Type: cross Abstract: Generative models produce many candidate solutions but provide no intrinsic signal by which to rank them. One mitigation is to demand a \emph{verifiable artifact}: a program that can be executed against the available evidence. A common concern is that requiring executable verification incurs costs in both solution quality and computational budget. We measure these two costs on the $120$-task ARC-AGI-2 evaluation set. Demanding a verifiable artifact from an LLM reduced the score from $93$ to $68$ of $120$ tasks ($77\%$ to $57\%$). Of those $25$ tasks, $21$ were reached by no program-producing configuration we ran, at any budget, by sampling or by search; yet for $27$ of the $28$ tasks that only the answer-only arm solved, some program arm held a program that reproduced every demonstration and still failed the test. The bottleneck is obtaining an explanation that \emph{generalizes}, not finding one. A short evolutionary search over two already-sampled programs, at no additional model cost, beats best-of-$2$ sampled code by $9.2 \pm 1.4$ tasks, and over a single sampled program beats $1$-shot sampling by $9.9 \pm 1.3$. Moreover, because a program can be checked against the training examples, any program-producing protocol can terminate early once a task is explained: under the same stopping rule, best-of-$n$ sampling declines up to $38\%$ of its budget and the evolved arm up to $49\%$, whereas an answer-only baseline cannot terminate early. Applying the same methodology to decoder synthesis for quantum error correction yields interpretable decoders evaluated against a certified panel of published decoders on held-out exam shots.

## TuringSpace: Grounding Spatial Intelligence in Action with Interactive CAPTCHAs

DevFeed: [TuringSpace: Grounding Spatial Intelligence in Action with Interactive CAPTCHAs](<https://devfeed.tech/articles/turingspace-grounding-spatial-intelligence-in-action-with-interactive-captchas-145049.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25387>)

Author: Xuwei Ding, Skylar Zhai, Linxin Song, Jiarui Feng, Dongqi Fu, Huajun Xi, Junnan Li, Jingcheng Liang, Jieyu Zhao, Yinglong Xia, Hong Li, Jian Kang

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Interactive AI Systems](<https://devfeed.tech/topics/interactive-ai-systems.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Hallucination detection](<https://devfeed.tech/topics/hallucination-detection.md>)

Tags: [3d](<https://devfeed.tech/tags/3d.md>), [actions](<https://devfeed.tech/tags/actions.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [captchas](<https://devfeed.tech/tags/captchas.md>), [clicks](<https://devfeed.tech/tags/clicks.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [grounding](<https://devfeed.tech/tags/grounding.md>), [interactive](<https://devfeed.tech/tags/interactive.md>), [it](<https://devfeed.tech/tags/it.md>), [k3](<https://devfeed.tech/tags/k3.md>), [kimi-k3](<https://devfeed.tech/tags/kimi-k3.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [qwen3](<https://devfeed.tech/tags/qwen3.md>), [qwen3-5](<https://devfeed.tech/tags/qwen3-5.md>), [real-world](<https://devfeed.tech/tags/real-world.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [results](<https://devfeed.tech/tags/results.md>), [spatial](<https://devfeed.tech/tags/spatial.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>), [tasks](<https://devfeed.tech/tags/tasks.md>), [test](<https://devfeed.tech/tags/test.md>)

### AI overview

TuringSpace tests spatial reasoning through action: its 125 synthetic, interactive CAPTCHAs require clicks and drags across five spatial abilities, with each task generated by a program that includes a checker and oracle. Humans solve 94.1%; GPT-6 Astra fails 25.9%, and open-weight Kimi K3 fails 68.0%. The authors attribute most model failures to misreading screenshots and report that reasoning often defends incorrect beliefs with hallucinated evidence. Fine-tuning Qwen3.5-9B on the tasks raises success from 1.6% to 33.1%, above Kimi K3, and improves results on all seven out-of-distribution benchmarks, including a 9.9-point gain on VSI-Bench indoor videos. The abstract reports transfer to real-world scenes; it does not establish broader scope beyond these stated evaluations.

### Source excerpt

arXiv:2611.25387v1 Announce Type: new Abstract: Humans act in the visual world by inferring the 3D layout behind a 2D view, predicting how an action changes it, and correcting themselves after seeing the result. Can multimodal large language models (MLLMs) do the same? CAPTCHAs offer a natural test, since humans solve them through this very loop, clicking and dragging on the screen and adjusting after each result. Yet existing CAPTCHA benchmarks are largely solved and rarely spatial, while spatial benchmarks rarely require action. We therefore present TuringSpace, a synthetic web environment of 125 interactive CAPTCHAs that span five spatial abilities and must be solved through clicks and drags on the screen. Every task is a program that generates new instances with a checker and an oracle, so the tasks also form a scalable training gym. Humans solve 94.1% of the tasks, whereas the best model, GPT-6 Astra, still fails 25.9% of them, and the best open-weight model, Kimi K3, fails 68.0%. Our further analysis shows that most failures come from misreading the screenshot, and that model reasoning rarely corrects a wrong belief but hallucinates evidence to defend it. To test whether such spatial abilities can be learned, we fine-tune Qwen3.5-9B on this gym. Its success rate rises from 1.6% to 33.1%, above Kimi K3, and it improves on all seven out-of-distribution benchmarks, including a 9.9-point gain on the real-world indoor videos of VSI-Bench. These results show that grounding spatial intelligence in screen actions exposes what MLLMs lack and offers a scalable way to teach it, with gains that transfer to real-world scenes. We will release TuringSpace to support further study.

## ForeSeek: Foresight-Driven Reasoning for Long-Form Video Understanding

DevFeed: [ForeSeek: Foresight-Driven Reasoning for Long-Form Video Understanding](<https://devfeed.tech/articles/foreseek-foresight-driven-reasoning-for-long-form-video-understanding-145624.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26024>)

Author: Yuchen Sun, Siyuan Cheng, Shi-Zhe Chen, Yixin Qin, Jinwen Luo, Zheng Wei

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [object-detection](<https://devfeed.tech/topics/object-detection.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [exploration](<https://devfeed.tech/tags/exploration.md>), [global](<https://devfeed.tech/tags/global.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [training](<https://devfeed.tech/tags/training.md>), [video](<https://devfeed.tech/tags/video.md>)

### AI overview

ForeSeek replaces reactive clip retrieval in long-form video agents with predictive exploration: it forms visually grounded expectations about unseen video segments, checks them through targeted observations, and refines its search as evidence emerges. Failed expectations guide backtracking. A two-stage training framework teaches multimodal language models foresight-guided reasoning and efficient decisions. Across five long-video benchmarks, the authors report competitive or better accuracy than the strongest search-based agent while observing 2.5-4.4 times fewer frames.

### Source excerpt

arXiv:2611.26024v1 Announce Type: new Abstract: Long-form video understanding, as a fundamental yet crucial vision task, is being revolutionized by multimodal large language model (MLLM)-based video agents. However, the retrieval-driven nature inherent to these agents creates a significant bottleneck, forcing them to reactively fetch clips based on local saliency without a global temporal plan. To address this, we present ForeSeek, a foresight-driven video agent that transforms long-video reasoning from reactive retrieval into predictive exploration. Rather than blindly probing the timeline, ForeSeek forms visually grounded expectations over unseen temporal regions, verifies them through targeted observations, and continuously refines its search based on emerging evidence, where failed expectations serve as informative signals for principled backtracking. Furthermore, we develop a two-stage training framework that equips MLLMs with foresight-guided reasoning and efficient decision making. Extensive experiments on five long-video benchmarks demonstrate that ForeSeek consistently improves the efficiency--accuracy trade-off of long-video reasoning, achieving competitive or superior accuracy while observing $2.5$--$4.4\times$ fewer frames than the strongest search-based agent. Code is available at https://github.com/harry0021/ForeSeek

## ROAM-GRPO: Enforcing Reasoning Diversity in GRPO via Routed Off-mode Approach Mining

DevFeed: [ROAM-GRPO: Enforcing Reasoning Diversity in GRPO via Routed Off-mode Approach Mining](<https://devfeed.tech/articles/roam-grpo-enforcing-reasoning-diversity-in-grpo-via-routed-off-mode-approach-mining-145610.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25926>)

Author: Jeff Duan, Matt Thomson

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [grpo](<https://devfeed.tech/topics/grpo.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [rlvr](<https://devfeed.tech/topics/rlvr.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [algorithm](<https://devfeed.tech/tags/algorithm.md>), [creativity](<https://devfeed.tech/tags/creativity.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [diversity](<https://devfeed.tech/tags/diversity.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [exploration](<https://devfeed.tech/tags/exploration.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [math](<https://devfeed.tech/tags/math.md>), [mining](<https://devfeed.tech/tags/mining.md>), [post-training](<https://devfeed.tech/tags/post-training.md>), [problems](<https://devfeed.tech/tags/problems.md>), [qwen3](<https://devfeed.tech/tags/qwen3.md>), [qwen3-5](<https://devfeed.tech/tags/qwen3-5.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [rlvr](<https://devfeed.tech/tags/rlvr.md>), [sft](<https://devfeed.tech/tags/sft.md>), [strategies](<https://devfeed.tech/tags/strategies.md>), [supervised-fine-tuning](<https://devfeed.tech/tags/supervised-fine-tuning.md>), [well](<https://devfeed.tech/tags/well.md>)

### AI overview

ROAM-GRPO aims to counter reasoning-mode collapse in GRPO by routing well-solved problems to an "off-mode" track. The model generates distinct solution strategies, an LLM judge checks their logical validity and distinctness, and accepted strategies are self-distilled into the policy. In the reported experiment, training Qwen3.5-2b on 1,000 MATH level-5 problems increased peak pass@16 by 4.7% and correct embedding cosine distance by approximately 1.5-2x on a difficult evaluation set. The supplied text does not detail the evaluation setup or broader limitations.

### Source excerpt

arXiv:2611.25926v1 Announce Type: new Abstract: Supervised fine-tuning (SFT) paired with Reinforcement Learning with Verifiable Rewards (RLVR) has become the dominant recipe for post-training LLMs on complex reasoning tasks, but has been shown to collapse the policy into a low-diversity mode. Repeated LLM outputs are nearly identical, and pass@k accuracy converges or degrades as a result. We propose ROAM-GRPO (Routed Off-mode Approach Mining for GRPO), an augmentation to the GRPO algorithm which incentivizes exploration of distinct reasoning strategies and aims to increase output creativity during evaluation. ROAM-GRPO detects well-solved problems during training and transfers them to an "off-mode" track. The model is prompted to produce distinct solution strategies to such problems, which are verified for logical validity and distinctness by an LLM judge and subsequently self-distilled into the policy. Training Qwen3.5-2b on a 1,000-problem subset of the MATH lvl5 dataset, ROAM-GRPO implementation increases peak pass@16 by 4.7% and correct embedding cosine distance by approximately 1.5-2x on a difficult evaluation set.

## MINTEval: Evaluating Memory under Multi-Target Interference in Long-Horizon Agent Systems

DevFeed: [MINTEval: Evaluating Memory under Multi-Target Interference in Long-Horizon Agent Systems](<https://devfeed.tech/articles/minteval-evaluating-memory-under-multi-target-interference-in-long-horizon-agent-systems-145334.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.18565>)

Author: Hyunji Lee, Justin Chih-Yao Chen, Joykirat Singh, Zaid Khan, Elias Stengel-Eskin, Mohit Bansal

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agent-frameworks](<https://devfeed.tech/tags/agent-frameworks.md>), [aggregation](<https://devfeed.tech/tags/aggregation.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [frameworks](<https://devfeed.tech/tags/frameworks.md>), [github](<https://devfeed.tech/tags/github.md>), [horizon](<https://devfeed.tech/tags/horizon.md>), [llms](<https://devfeed.tech/tags/llms.md>), [long-context](<https://devfeed.tech/tags/long-context.md>), [long-context-llms](<https://devfeed.tech/tags/long-context-llms.md>), [memory](<https://devfeed.tech/tags/memory.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [recall](<https://devfeed.tech/tags/recall.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [revisions](<https://devfeed.tech/tags/revisions.md>), [robustness](<https://devfeed.tech/tags/robustness.md>), [state](<https://devfeed.tech/tags/state.md>), [tasks](<https://devfeed.tech/tags/tasks.md>)

### AI overview

MINTEval tests how seven systems--including long-context LLMs, RAG, and memory-augmented agents--handle evolving, interference-heavy information. Its 15.6k question-answer pairs span four domains and contexts averaging 138.8k tokens, up to 1.8M. Tasks test retrieval of a single target and aggregation across multiple pieces of evidence. Average accuracy was 27.9%, with multi-evidence reasoning especially difficult. The authors identify retrieval and memory construction as primary limitations; systems also struggled with earlier facts revised or disrupted by later context, and accuracy fell as intervening updates increased.

### Source excerpt

arXiv:2605.18565v3 Announce Type: replace-cross Abstract: Real-world agents operate over long and evolving horizons, where information is repeatedly updated and may interfere across memories, requiring accurate recall and aggregated reasoning over multiple pieces of information. However, existing benchmarks focus on static, independent recall and fail to capture these dynamic interactions between evolving memories. In this paper, we study how current memory-augmented agents perform in realistic, interference-heavy, long-horizon settings across diverse domains and question types. We introduce MINTEval (Long-Horizon Memory under INTerference Evaluation), a benchmark featuring (1) long, highly interconnected contexts with frequently updated information that induces substantial interference, (2) diverse domains (state tracking, multi-turn dialogue, Wikipedia revisions, and GitHub commits), enabling evaluation of domain generalization, and (3) diverse question types that assess robustness to interference, including (i) single-target recall tasks requiring retrieval of a specific target from long contexts, and (ii) multi-target aggregation tasks requiring reasoning over multiple relevant pieces of information. Overall, MINTEval has 15.6k question-answering pairs over long-horizon contexts averaging 138.8k tokens and extending up to 1.8M tokens per instance. We evaluate 7 representative systems, including vanilla long-context LLMs, RAG, and memory-augmented agent frameworks. Across all systems, we observe consistently low performance (avg. 27.9% accuracy), especially on questions requiring aggregated reasoning over multiple pieces of evidence. Our analysis shows that performance is primarily limited by retrieval and memory construction. Furthermore, current memory systems struggle to recall and reason over earlier facts that are revised or interfered with by subsequent context, with accuracy degrading as the number of intervening updates increases.

## When and How Much to Imagine: Adaptive Test-Time Scaling with World Models for Visual Spatial Reasoning

DevFeed: [When and How Much to Imagine: Adaptive Test-Time Scaling with World Models for Visual Spatial Reasoning](<https://devfeed.tech/articles/when-and-how-much-to-imagine-adaptive-test-time-scaling-with-world-models-for-visual-spatial-reasoning-145315.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2602.08236>)

Author: Shoubin Yu, Yue Zhang, Zun Wang, Jaehong Yoon, Huaxiu Yao, Mingyu Ding, Mohit Bansal

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [World models](<https://devfeed.tech/topics/world-models.md>), [grpo](<https://devfeed.tech/topics/grpo.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [control](<https://devfeed.tech/tags/control.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [imagination](<https://devfeed.tech/tags/imagination.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [spatial-reasoning](<https://devfeed.tech/tags/spatial-reasoning.md>), [world-models](<https://devfeed.tech/tags/world-models.md>)

### AI overview

Visual imagination can help spatial reasoning when a scene must be assessed from an unseen viewpoint, but unnecessary or excessive imagined evidence can waste computation or reduce accuracy. The paper introduces AVIC, which checks whether current visual evidence is sufficient before selectively invoking and scaling a world model, and AVIC-R, which learns that policy using GRPO with answer-correctness rewards and imagination-cost penalties, without labels for when or how much to imagine. On SAT, MMSI, and R2R, selective control reportedly matches or outperforms fixed imagination strategies with fewer world-model calls and language tokens. AVIC-R also reportedly surpasses GPT-4o and GPT-4.1 while using the world model less often. The supplied abstract does not provide quantitative results or benchmark-specific details.

### Source excerpt

arXiv:2602.08236v3 Announce Type: replace-cross Abstract: Despite rapid progress in MLLMs, visual spatial reasoning remains unreliable when correct answers depend on how a scene would appear under unseen or alternative viewpoints. Recent work addresses this by augmenting reasoning with world models for visual imagination, but questions such as when imagination is actually necessary, how much of it is beneficial, and when it becomes harmful, remain poorly understood. In practice, indiscriminate imagination can increase computation and even degrade performance by introducing misleading evidence. In this work, we present an in-depth analysis of test-time visual imagination as a controllable resource for spatial reasoning. We first study when static visual evidence is sufficient, when imagination improves reasoning, and how excessive or unnecessary imagination affects accuracy and efficiency. To support this analysis, we then introduce AVIC, an adaptive test-time framework with world models that explicitly reasons about the sufficiency of current visual evidence before selectively invoking and scaling visual imagination. Finally, to further learn this gating and planning behavior without any annotation of when and how much to imagine, we introduce AVIC-R, which trains the policy via GRPO from QA-correctness rewards and penalties by imagination cost. Across spatial reasoning benchmarks (SAT, MMSI) and an embodied navigation benchmark (R2R), our results reveal clear scenarios where imagination is critical, marginal, or detrimental, and show that selective control can match or outperform fixed imagination strategies with substantially fewer world-model calls and language tokens. Our AVIC-R surpasses strong proprietary baselines including GPT-4o and GPT-4.1 while invoking the world model less often. Overall, our findings highlight the importance of analyzing and controlling test-time imagination for efficient and reliable spatial reasoning.

## ClosureBench: A Benchmark for Reasoning under Closure Assumptions

DevFeed: [ClosureBench: A Benchmark for Reasoning under Closure Assumptions](<https://devfeed.tech/articles/closurebench-a-benchmark-for-reasoning-under-closure-assumptions-145657.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26335>)

Author: Francesco Belardinelli, Flavio Corradini, Matteo Leonesi

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [ai](<https://devfeed.tech/tags/ai.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [choice](<https://devfeed.tech/tags/choice.md>), [closure](<https://devfeed.tech/tags/closure.md>), [computer-science](<https://devfeed.tech/tags/computer-science.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [evaluations](<https://devfeed.tech/tags/evaluations.md>), [extensions](<https://devfeed.tech/tags/extensions.md>), [information](<https://devfeed.tech/tags/information.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [owa](<https://devfeed.tech/tags/owa.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [records](<https://devfeed.tech/tags/records.md>), [rules](<https://devfeed.tech/tags/rules.md>), [scope](<https://devfeed.tech/tags/scope.md>), [source](<https://devfeed.tech/tags/source.md>), [tests](<https://devfeed.tech/tags/tests.md>), [updates](<https://devfeed.tech/tags/updates.md>)

### AI overview

ClosureBench evaluates whether LLM agents apply stated assumptions about missing information: open-world (OWA), closed-world (CWA), or locally closed-world (LCWA). Each base scenario is tested in all three variants, changing only how missing information is treated; a base scenario counts as correct only when answers are correct across every variant. The benchmark also includes extensions for action choice, source scope, and updates. Across 960 instances, overall accuracy ranged from 71.18% to 99.86%, while accuracy on cases where the correct label changes with the assumption ranged from 18.23% to 99.31%. The abstract reports notable errors under LCWA, including agents answering "unknown" when missing information should be treated as false. These results suggest overall accuracy can obscure failures to follow the specified closure assumption.

### Source excerpt

arXiv:2611.26335v1 Announce Type: new Abstract: AI agents built on large language models (LLMs) often answer questions from incomplete records. Missing information might be interpreted as unknown or false, depending on the completeness of the source. Current evaluations fail to distinguish these two aspects, as they are usually interwoven with other reasoning demands. We present ClosureBench, a benchmark that tests whether LLMs follow the stated rules for dealing with missing information. Every base scenario is tested under OWA, CWA, and LCWA, and only the treatment of missing information changes between variants. ClosureBench provides a verified base benchmark as well as agent extensions for action choice, source scope, and updates. A model is considered correct on a base scenario only if it provides correct answers in all of its closure variants. Across 960 instances, overall accuracy can mask substantial closure mistakes. Models achieve overall accuracy from 71.18% to 99.86% across replicated runs, but their core switch accuracy, measured on the scenarios whose gold label actually changes with the contract, varies between 18.23% and 99.31%. Large discrepancies emerge under LCWA. For example, some agents choose unknown despite the prompt indicating that missing information should be treated as false. Our findings demonstrate that evaluations of AI agents should not only measure whether agents can reason over existing facts but also whether they use the appropriate assumption when information is absent.

## Realized versus Population Reliability: Anytime-Valid Self-Consistency Certification under Correlated Reasoning

DevFeed: [Realized versus Population Reliability: Anytime-Valid Self-Consistency Certification under Correlated Reasoning](<https://devfeed.tech/articles/realized-versus-population-reliability-anytime-valid-self-consistency-certification-under-correlated-reasoning-145197.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26592>)

Author: Terence D. Agbeyegbe, Toneukarin G. Agbeyegbe

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Uncertainty quantification LLMs](<https://devfeed.tech/topics/uncertainty-quantification-llms.md>)

Tags: [aime](<https://devfeed.tech/tags/aime.md>), [betting](<https://devfeed.tech/tags/betting.md>), [certificate](<https://devfeed.tech/tags/certificate.md>), [certification](<https://devfeed.tech/tags/certification.md>), [claude-haiku-4-5](<https://devfeed.tech/tags/claude-haiku-4-5.md>), [consistency](<https://devfeed.tech/tags/consistency.md>), [correlation](<https://devfeed.tech/tags/correlation.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [error-control](<https://devfeed.tech/tags/error-control.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [prompts](<https://devfeed.tech/tags/prompts.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [simulations](<https://devfeed.tech/tags/simulations.md>), [stat-me](<https://devfeed.tech/tags/stat-me.md>)

### AI overview

Correlated language-model rollouts change what an anytime-valid self-consistency certificate establishes. Under a per-prompt latent-state model, the standard betting e-process controls error for the realized belief, but not necessarily for average population behavior; the two null hypotheses coincide only at zero correlation. Within-prompt rollout averages have effective sample size capped at 1/ρ, so adding rollouts cannot repair population validity. A prompt-level e-process restores it, and under a pure rollout budget the reported optimal allocation is one rollout per prompt. In Claude Haiku 4.5 experiments, the gap was dormant on GSM8K (estimated ρ=0.435, θ=+0.96), but on AIME (ρ=0.480, θ=−0.15) the single-unit rule crossed on 24% of prompts at nominal ε=0.05, while the prompt-level procedure declined (0.00).

### Source excerpt

arXiv:2611.26592v1 Announce Type: cross Abstract: Self-consistency certification treats repeated sampling from a language model as sequential evidence accumulation and stops when an anytime-valid e-process crosses a threshold, guaranteeing finite-sample Type I error control by Ville's inequality. Prior constructions assume sampled rollouts are conditionally independent. We show this assumption fails in practice: reasoning streams are positively correlated through shared reasoning prefixes or across-prompt heterogeneity, changing not merely certificate efficiency but the hypothesis certified. Modeling dependence via a per-unit latent state (the random-effects / intraclass-correlation structure) induces exchangeable, equi-correlated increments with correlation $\rho$. Correlation splits the single null into a realized null about the drawn belief and a population null about average model behavior, which coincide only when $\rho=0$. We show: (i) the standard betting e-process remains anytime-valid for the realized null, not the population null; (ii) read as a population-null test, a suitable mixture has asymptotic crossing probability at least $\pi=P(\mu(Z)>0)$, potentially exceeding nominal $\epsilon$; (iii) the within-unit mean's effective sample size saturates at $1/\rho$, so extra within-unit rollouts cannot restore population validity; and (iv) a prompt-level e-process restores population validity, with optimal allocation at one rollout per prompt under a pure rollout budget. Simulations and Claude Haiku 4.5 rollouts confirm this regime dependence: on GSM8K ($\hat{\rho}=0.435$, $\hat{\theta}=+0.96$) the gap is dormant, whereas on AIME ($\hat{\rho}=0.480$, $\hat{\theta}=-0.15$) the single-unit rule crosses on 24% of prompts at nominal $\epsilon=0.05$, while the prompt-level procedure declines (0.00). This correctness result identifies which reliability claim an anytime-valid certificate actually supports under dependence.

## Many Solutions, One Answer: Search Order Shapes the Default Solutions of Language Models

DevFeed: [Many Solutions, One Answer: Search Order Shapes the Default Solutions of Language Models](<https://devfeed.tech/articles/many-solutions-one-answer-search-order-shapes-the-default-solutions-of-language-models-145609.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25916>)

Author: Qinhan Hou, Jing Tang

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Chain-of-thought](<https://devfeed.tech/topics/chain-of-thought.md>), [Open Source Models & Datasets](<https://devfeed.tech/topics/open-source-models-datasets.md>)

Tags: [constraint](<https://devfeed.tech/tags/constraint.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [models](<https://devfeed.tech/tags/models.md>), [open-weight-models](<https://devfeed.tech/tags/open-weight-models.md>), [order](<https://devfeed.tech/tags/order.md>), [puzzle](<https://devfeed.tech/tags/puzzle.md>), [puzzles](<https://devfeed.tech/tags/puzzles.md>), [random](<https://devfeed.tech/tags/random.md>), [responses](<https://devfeed.tech/tags/responses.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [search](<https://devfeed.tech/tags/search.md>), [solutions](<https://devfeed.tech/tags/solutions.md>)

### AI overview

Across four open-weight language models, repeated answers to multi-solution puzzles cluster around a shared default solution far more than uniform sampling would predict. The models can still produce less common solutions when clues uniquely determine them. On held-out symbolic puzzles, a search visiting entities with the fewest initial candidates first better matched responses than name-order search when their predictions differed, by 26-40 percentage points. Changing value or search order shifted answers toward the favored solution. Randomized presentation or search order increased solution breadth by 26-42 points on symbolic puzzles and 11-26 on ZebraLogic, with maximum observed validity losses of 3 and 4 points. Randomized presentation broadened answers more than a random-choice instruction in each of the three main models.

### Source excerpt

arXiv:2611.25916v1 Announce Type: cross Abstract: When a constraint puzzle has several correct answers, language models tend to give the same one.We study four open-weight models from three families on symbolic puzzles and multi-solution ZebraLogic variants with exactly known solution sets.Repeated samples concentrate on one default solution far beyond uniform sampling, and different models often share it.Yet models usually produce even rarely sampled solutions when added clues uniquely identify them.On held-out symbolic puzzles, we compare visiting entities in name order with visiting those with the fewest initial candidates first. Both searches try values in the stated order. Where their pre-specified predictions differ, the latter matches a larger share of responses in all four models, by 26 to 40 percentage points.Reversing the stated value order or specifying a search order shifts responses towards the solution that the new order favours, on both testbeds. Randomising the presentation or supplying a random search order raises breadth (the effective share of solutions in use) by 26 to 42 percentage points on the symbolic task and 11 to 26 on ZebraLogic. The largest observed validity losses are 3 and 4 percentage points, respectively. Randomised presentation broadens answers more than a random-choice instruction in each of the three main models.

## Beam Model Technical Report

DevFeed: [Beam Model Technical Report](<https://devfeed.tech/articles/beam-model-technical-report-145663.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26358>)

Author: Reflection Team, Adam Abdulhamid, Osama Abuelsorour, Yasmin Alameddine, Carlo Alfano, Duarte Alves, Brandon Amos, Selvaraj Anandaraj, Arseni Anisimovich, George Antoniadis, Ioannis Antonoglou, Sheila

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [context window](<https://devfeed.tech/topics/context-window.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [Model Optimizer](<https://devfeed.tech/topics/model-optimizer.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>)

Tags: [agentic-workloads](<https://devfeed.tech/tags/agentic-workloads.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [beam](<https://devfeed.tech/tags/beam.md>), [co-design](<https://devfeed.tech/tags/co-design.md>), [coding](<https://devfeed.tech/tags/coding.md>), [context-window](<https://devfeed.tech/tags/context-window.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [fp8](<https://devfeed.tech/tags/fp8.md>), [gb300](<https://devfeed.tech/tags/gb300.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [nvfp4](<https://devfeed.tech/tags/nvfp4.md>), [open](<https://devfeed.tech/tags/open.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [speculative-decoding](<https://devfeed.tech/tags/speculative-decoding.md>), [throughput](<https://devfeed.tech/tags/throughput.md>)

### AI overview

Beam is a 501.3B-parameter sparse MoE model with 23.4B active parameters, designed for coding, reasoning, and agentic workloads. The abstract reports a 1M-token context window enabled by two-stage midtraining on verifiable, correctness-anchored data, following pretraining on 23.8T tokens. Post-training used asynchronous RL on 10.5K GB300 GPUs, producing over 100M rollouts in four weeks; measured capabilities continued to improve as RL compute increased, with no plateau reported. Hardware-model co-design, FP8/NVFP4 quantization, and a 1.93B-parameter DFlash 2 speculative decoding drafter target serving throughput. The report claims 3-4x less inference compute than many comparable models. On its safety evaluation, Beam refused or pivoted on 88.9% of harmful requests while complying with 81.5% of harmless requests; the text says its harmful-request refusal/pivot rate was highest among open-weight models on the Pareto frontier.

### Source excerpt

arXiv:2611.26358v1 Announce Type: cross Abstract: Beam is a sparse Mixture-of-Experts (MoE) model with 501.3B total parameters and 23.4B active parameters, specifically built and optimized for coding, reasoning, and agentic workloads. Beam delivers frontier level reasoning, while requiring 3-4x less inference compute than many models of its class. Pretrained on 23.8T diverse, curated, high-quality tokens, Beam features a 1M-token context window achieved through a novel two-stage midtraining phase, built on verifiable, correctness-anchored data. Post-training leveraged a large-scale, fully asynchronous reinforcement learning (RL) campaign across 10.5K GB300 GPUs, generating over 100M rollouts over 4 weeks of training. We believe this is one of the largest scale RL runs conducted by any open lab to date. Across our evaluation suite, capabilities continued to improve as we increased RL compute, with no sign of a plateau. Supported by deep hardware-model co-design, FP8/NVFP4 quantization, and a 1.93B-parameter speculative decoding drafter (DFlash 2), Beam maximizes serving throughput with efficient, flexible deployment. Averaged across conversation turns and harm categories, Beam refuses or pivots on 88.9% of harmful requests while complying with 81.5% of harmless requests, achieving the highest refusal/pivot rate on harmful requests among open-weight models on the Pareto frontier.

## LLM4Rail: An LLM-Augmented Personalized Railway Services Platform

DevFeed: [LLM4Rail: An LLM-Augmented Personalized Railway Services Platform](<https://devfeed.tech/articles/llm4rail-an-llm-augmented-personalized-railway-services-platform-145306.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2507.23377>)

Author: Zhuo Li, Xianghuai Deng, Chiwei Feng, Hanmeng Li, Shenjie Wang, Junchao Fan, Jiaxing Shen, Zhiyuan Wen, Longxiang Gao, Linchun Wu

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [recommendation systems](<https://devfeed.tech/topics/recommendation-systems.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [function calling](<https://devfeed.tech/topics/function-calling.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>)

Tags: [alignment](<https://devfeed.tech/tags/alignment.md>), [arguments](<https://devfeed.tech/tags/arguments.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [food](<https://devfeed.tech/tags/food.md>), [framework](<https://devfeed.tech/tags/framework.md>), [generation](<https://devfeed.tech/tags/generation.md>), [generative](<https://devfeed.tech/tags/generative.md>), [language](<https://devfeed.tech/tags/language.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llms](<https://devfeed.tech/tags/llms.md>), [mechanism](<https://devfeed.tech/tags/mechanism.md>), [modules](<https://devfeed.tech/tags/modules.md>), [novel](<https://devfeed.tech/tags/novel.md>), [platform](<https://devfeed.tech/tags/platform.md>), [railway](<https://devfeed.tech/tags/railway.md>), [service](<https://devfeed.tech/tags/service.md>), [services](<https://devfeed.tech/tags/services.md>), [similarity](<https://devfeed.tech/tags/similarity.md>), [system](<https://devfeed.tech/tags/system.md>)

### AI overview

LLM4Rail is a personalized railway-services platform with modules for ticket and weather inquiries, onboard food and drink recommendations, and casual conversation. Its iterative ER-QTAO framework uses Experience Storage and Retrieval and Diagnostic Observation Generation to help the LLM call external tools with complex arguments. For catering, the authors introduce the Chinese Railway Food & Drink dataset, organized by city, cuisine, age group, and spiciness, and a zero-shot conversational recommender. A feature-similarity alignment mechanism is intended to keep generated recommendations matched to dataset entries.

### Source excerpt

arXiv:2507.23377v2 Announce Type: replace Abstract: Large language models (LLMs) have significantly reshaped different walks of business. To meet the increasing demands for individualized railway service, we develop LLM4Rail - a novel LLM-augmented personalized railway services platform. Empowered by the LLM, LLM4Rail provides dedicated modules for ticket and weather inquiries, food & drink recommendations, and casual conversation. In LLM4Rail, we propose the iterative "Experience-Retrieval Question-Thought-Action-Observation (ER-QTAO)" reasoning framework. It introduces two key mechanisms, Experience Storage and Retrieval (ESR) and Diagnostic Observation Generation (DOG), to enhance the LLM's ability to invoke appropriate external tools with complex arguments. To provide personalized onboard dining services, we first construct the Chinese Railway Food & Drink (CRFD) dataset, which covers a wide range of signature dishes categorized by city, cuisine, age group, and spiciness level, and then develop an LLM-based zero-shot conversational recommender system for railway catering. In addition, to address the unconstrained nature of generative recommendations, we design a feature-similarity-based alignment mechanism to ensure that all recommended items are aligned with entries in the CRFD dataset.

## Incorporating argumentation frameworks for LLM decision-making: Task-specific design and evaluation

DevFeed: [Incorporating argumentation frameworks for LLM decision-making: Task-specific design and evaluation](<https://devfeed.tech/articles/incorporating-argumentation-frameworks-for-llm-decision-making-task-specific-design-and-evaluation-145502.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25361>)

Author: Bochen Han, Anshu Xiong, Mingyang Ma

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [decision-making](<https://devfeed.tech/topics/decision-making.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [arguments](<https://devfeed.tech/tags/arguments.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [decision-making](<https://devfeed.tech/tags/decision-making.md>), [decision-models](<https://devfeed.tech/tags/decision-models.md>), [design](<https://devfeed.tech/tags/design.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [fact-verification](<https://devfeed.tech/tags/fact-verification.md>), [formal](<https://devfeed.tech/tags/formal.md>), [frameworks](<https://devfeed.tech/tags/frameworks.md>), [generative](<https://devfeed.tech/tags/generative.md>), [grounded](<https://devfeed.tech/tags/grounded.md>), [inference](<https://devfeed.tech/tags/inference.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [llms](<https://devfeed.tech/tags/llms.md>), [model-selection](<https://devfeed.tech/tags/model-selection.md>), [semantic](<https://devfeed.tech/tags/semantic.md>), [semantics](<https://devfeed.tech/tags/semantics.md>), [tasks](<https://devfeed.tech/tags/tasks.md>)

### AI overview

Argumentation frameworks do not reliably improve LLM decision-making across tasks; their design needs to fit the task. For debate-based decisions, Qnt-Arg groups arguments by original speaker and combines estimated argument strengths with candidate-answer priors using gradual semantics. It consistently outperformed other argumentation-augmented methods in matched configurations. For retrieval-based fact verification, Qlt-Arg uses credibility estimates to rank conflicting evidence, then grounded semantics to select evidence for LLM inference; it improved performance over the baselines. The study also examines how generative and decision models contribute to these methods. These findings are limited to the evaluated tasks and configurations.

### Source excerpt

arXiv:2611.25361v1 Announce Type: cross Abstract: Argumentation frameworks (AFs) provide explicit, formal mechanisms for representing and resolving conflicts and thus have been incorporated into large language model (LLM) decision-making. The benefits vary and depend on how arguments and their relations are constructed, evaluated, and incorporated into the decision process. {We abstract a workflow consisting of four stages based on existing argumentation-augmented methods} and investigate their design choices over tasks of debate-based decision-making and retrieval-based fact verification. Our evaluation with recent LLMs shows that existing methods do not consistently improve performance across the considered tasks. Motivated by this observation, we develop two task-specific methods. The first is called Qnt-Arg, which organizes arguments around the original speakers in debate materials and aggregates model-estimated argument strengths and candidate-answer priors under gradual semantics. The second is Qlt-Arg that uses credibility estimates to establish preferences between conflicting evidence items and applies grounded semantics to select acceptable evidence for subsequent LLM inference. We examine the roles of generative and decision models within the two methods. In the debate setting, Qnt-Arg consistently outperforms other argumentation-augmented methods across matched configurations; Qlt-Arg improves retrieval-based fact-verification performance compared with the baselines. These findings demonstrate the value of combining formal argumentation and LLM while highlighting the importance of aligning with tasks in argumentation construction, model selection, and the role of formal semantic reasoning.

## MatVerse: A Foundation Model for Materials Science

DevFeed: [MatVerse: A Foundation Model for Materials Science](<https://devfeed.tech/articles/matverse-a-foundation-model-for-materials-science-145560.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25651>)

Author: Xuanyu Zhu, Qian Tan, Bo Chen, Jiaqing Xie, Yang Han, Mei Wu, Wanhao Liu, Xuanze Lin, Yuchen Zhou, Shijie Wang, Biqing Qi, Shixiang Tang, Wenwei Zhang, Tianfan Fu, Kai Yu, Lu Chen, Yuqiang Li, Xin Che

Published: 2026-11-20T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Materials science and engineering](<https://devfeed.tech/topics/materials-science-and-engineering.md>), [AI for science](<https://devfeed.tech/topics/ai-for-science.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-assisted](<https://devfeed.tech/tags/ai-assisted.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [capabilities](<https://devfeed.tech/tags/capabilities.md>), [cond-mat-mtrl-sci](<https://devfeed.tech/tags/cond-mat-mtrl-sci.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [foundation-model](<https://devfeed.tech/tags/foundation-model.md>), [inference-efficiency](<https://devfeed.tech/tags/inference-efficiency.md>), [materials](<https://devfeed.tech/tags/materials.md>), [materials-science](<https://devfeed.tech/tags/materials-science.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [post-training](<https://devfeed.tech/tags/post-training.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [vision](<https://devfeed.tech/tags/vision.md>)

### AI overview

MatVerse is a materials-science foundation model built on Intern-S2-Preview and specialized through large-scale domain post-training. Its multimodal corpus covers materials knowledge, scientific reasoning, quantitative problems, structures, and scientific interpretation; data preparation combines teacher-guided reasoning distillation, synthesized problems, consistency filtering, and retrieval-based multimodal construction. The report says MatVerse improves over the base model on OmniMatBench, MatSciBench, and MaCBench, and improves inference efficiency on calculation-intensive problems. The excerpt gives no numerical results or detailed benchmark comparisons.

### Source excerpt

arXiv:2611.25651v1 Announce Type: new Abstract: Materials science involves highly heterogeneous scientific information, including textual knowledge, molecular and crystal structures, experimental characterization data, microscopy images, and structure-property relationships. Developing a general-purpose foundation model capable of understanding and reasoning over these diverse materials representations remains challenging. In this technical report, we introduce MatVerse, a materials foundation model built upon Intern-S2-Preview and further specialized through large-scale materials-domain post-training. We construct a large-scale multimodal training corpus covering materials knowledge, scientific reasoning, quantitative problem solving, structural understanding, and multimodal scientific interpretation. To improve domain adaptation, we develop a unified data construction pipeline combining teacher-guided reasoning distillation, scientific problem synthesis, consistency-based filtering, and retrieval-based multimodal data construction. We evaluate MatVerse on three representative materials-science benchmarks: our human-calibrated OmniMatBench, MatSciBench, and MaCBench. Results demonstrate consistent improvements after materials-specific post-training over the base model on all three benchmarks, alongside improved inference efficiency on calculation-intensive problems. These results show that targeted domain-specific post-training can effectively enhance scientific reasoning, materials knowledge understanding, and multimodal scientific capabilities of foundation models. MatVerse provides a unified language-and-vision interface for materials science and serves as a foundation for future AI-assisted materials research workflows.

## From "Weak" Signals to Strong Models: Preference Delta Aggregation with LoRA Merging

DevFeed: [From "Weak" Signals to Strong Models: Preference Delta Aggregation with LoRA Merging](<https://devfeed.tech/articles/from-weak-signals-to-strong-models-preference-delta-aggregation-with-lora-merging-145341.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2606.00357>)

Author: Qi Sun, Siyue Zhang, Yulin Chen, Yuxiang Xue, Ru Peng, Chen Zhao

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [lora](<https://devfeed.tech/topics/lora.md>), [rlhf](<https://devfeed.tech/topics/rlhf.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [qwen](<https://devfeed.tech/topics/qwen.md>)

Tags: [adapter](<https://devfeed.tech/tags/adapter.md>), [adapters](<https://devfeed.tech/tags/adapters.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [aggregation](<https://devfeed.tech/tags/aggregation.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [capabilities](<https://devfeed.tech/tags/capabilities.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [data](<https://devfeed.tech/tags/data.md>), [delta](<https://devfeed.tech/tags/delta.md>), [evaluations](<https://devfeed.tech/tags/evaluations.md>), [knowledge](<https://devfeed.tech/tags/knowledge.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [llms](<https://devfeed.tech/tags/llms.md>), [lora](<https://devfeed.tech/tags/lora.md>), [models](<https://devfeed.tech/tags/models.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [pairs](<https://devfeed.tech/tags/pairs.md>), [pda](<https://devfeed.tech/tags/pda.md>), [qwen3](<https://devfeed.tech/tags/qwen3.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>)

### AI overview

Preference Delta Aggregation (PDA) turns preference comparisons between weaker models into separate LoRA adapters trained through preference optimization, then merges them to improve a stronger model. Its Geometric Alignment Merging (GAM) aligns the adapters' left and right singular bases with orthogonal Procrustes rotations before averaging their factors. In evaluations on knowledge reasoning and agentic search, PDA-GAM outperformed the strongest individual delta. Three reasoning deltas and four agentic deltas improved Qwen3-8B's average scores by 6.8 and 7.3 points, respectively--2.1 and 4.3 points above the strongest single-delta baselines. The abstract reports similar gains on a second student model and attributes the results to combining complementary capabilities.

### Source excerpt

arXiv:2606.00357v3 Announce Type: replace Abstract: Training strong large language models (LLMs) requires high-quality supervision, which is often scarce. Recent work shows that paired preference data from weak-weaker model pairs (e.g., Qwen3 4B over 1.7B), despite the limited quality of individual responses, can provide an effective supervision signal through relative quality deltas, which we term a "weak" signal. This motivates a key research question: can multiple "weak" signals be constructively aggregated for improving strong models (e.g., Qwen3 8B)? To this end, we propose Preference Delta Aggregation (PDA), the first framework that derives a preference delta from each weak-weaker model pair, instantiates it as a LoRA adapter learned through preference optimization, and aggregates the resulting deltas via LoRA merging. To improve the composition of independently trained adapters, we introduce Geometric Alignment Merging (GAM), which aligns their left and right singular bases through orthogonal Procrustes rotations and averages the aligned factors. Evaluations on knowledge reasoning and agentic search show that PDA-GAM exceeds the strongest individual delta, with further gains along the evaluated sequences of additional deltas. Using three reasoning deltas and four agentic deltas, PDA-GAM improves Qwen3-8B by 6.8 and 7.3 points on average, respectively, exceeding the strongest single-delta baseline by 2.1 and 4.3 points. These configurations achieve the highest average scores among the evaluated merging methods, with similar gains observed on a second student model. Further analysis suggests that these gains come from composing complementary capabilities encoded across distinct preference deltas. Code is available at https://github.com/AlbertQiSun/Preference-Delta-Aggregation.

## NL2AGBench: Benchmarking LLM Auto-Formalization for AlphaGeometry

DevFeed: [NL2AGBench: Benchmarking LLM Auto-Formalization for AlphaGeometry](<https://devfeed.tech/articles/nl2agbench-benchmarking-llm-auto-formalization-for-alphageometry-145365.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2608.28481>)

Author: Samuel Xiao, Judy Song, Rory Hu, Ziliang Zong

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Language Engineering](<https://devfeed.tech/topics/language-engineering.md>), [domain-specific language](<https://devfeed.tech/topics/domain-specific-language.md>), [Enhancing AI model performance](<https://devfeed.tech/topics/enhancing-ai-model-performance.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [machine-translation](<https://devfeed.tech/topics/machine-translation.md>)

Tags: [arxiv](<https://devfeed.tech/tags/arxiv.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [domain-specific-language](<https://devfeed.tech/tags/domain-specific-language.md>), [error](<https://devfeed.tech/tags/error.md>), [execution](<https://devfeed.tech/tags/execution.md>), [few-shot](<https://devfeed.tech/tags/few-shot.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [llm](<https://devfeed.tech/tags/llm.md>), [open-source-models](<https://devfeed.tech/tags/open-source-models.md>), [performance](<https://devfeed.tech/tags/performance.md>), [source](<https://devfeed.tech/tags/source.md>), [taxonomy](<https://devfeed.tech/tags/taxonomy.md>), [translation](<https://devfeed.tech/tags/translation.md>)

### AI overview

NL2AGBench evaluates how well LLMs translate English geometry problems into AlphaGeometry's specialized DSL, using execution within AlphaGeometry to assess whether translations work rather than relying only on textual similarity. Across ten open and closed models at multiple scales, leading closed models exceed 80% executable translation, while even the largest open models struggle to preserve geometric constraints and produce valid formalizations. The benchmark distinguishes syntax from logic errors and reports measurable improvements from few-shot prompting, fine-tuning, and human-guided hints across multiple model families.

### Source excerpt

arXiv:2608.28481v2 Announce Type: replace-cross Abstract: Recent advances in large language models (LLMs) have demonstrated strong capabilities in natural language understanding and mathematical reasoning. However, their ability to translate informal mathematical problems into formal representations remains underexplored. This limitation is particularly important for neuro-symbolic geometry systems such as AlphaGeometry, whose theorem-proving engine requires inputs in a specialized domain-specific language (DSL). Although AlphaGeometry achieves near-IMO gold-medalist performance, manually converting natural-language problems into its formal syntax remains a significant usability bottleneck. To address this challenge, we introduce the Natural Language to AlphaGeometry Benchmark (NL2AGBench), which evaluates LLMs in translating English geometry problems into AlphaGeometry-compatible formal representations. NL2AGBench uses execution-based verification within AlphaGeometry to assess translation quality rather than relying solely on textual similarity. We evaluate ten state-of-the-art open- and closed-source LLMs across multiple parameter scales and analyze executable translation accuracy, syntactic correctness, and error characteristics. Our experiments reveal a substantial performance gap between closed- and open-source models: leading closed-source models achieve executable translation rates above 80%, while even the largest open-source models struggle to consistently preserve geometric constraints and produce valid formalizations. We introduce an error taxonomy distinguishing syntax and logic errors and investigate mitigation strategies, including few-shot prompting, fine-tuning, and human-guided hinting, which yield measurable improvements across multiple model families.

## GLiDE: A Fast, Calibrated Decision Model Distilled from Reasoning Teachers

DevFeed: [GLiDE: A Fast, Calibrated Decision Model Distilled from Reasoning Teachers](<https://devfeed.tech/articles/glide-a-fast-calibrated-decision-model-distilled-from-reasoning-teachers-145678.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26509>)

Author: Sahibzada Allahyar, Julia White, Dhruv Atreja, George Hurn-Maloney, Urchade Zaratiana

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Training AI Models](<https://devfeed.tech/topics/training-ai-models.md>), [Post-training optimization](<https://devfeed.tech/topics/post-training-optimization.md>), [model architecture](<https://devfeed.tech/topics/model-architecture.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [qwen](<https://devfeed.tech/topics/qwen.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [calibrated-decision-model](<https://devfeed.tech/tags/calibrated-decision-model.md>), [confidence](<https://devfeed.tech/tags/confidence.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [decision](<https://devfeed.tech/tags/decision.md>), [decision-model](<https://devfeed.tech/tags/decision-model.md>), [decision-models](<https://devfeed.tech/tags/decision-models.md>), [di](<https://devfeed.tech/tags/di.md>), [distillation](<https://devfeed.tech/tags/distillation.md>), [fast](<https://devfeed.tech/tags/fast.md>), [inference](<https://devfeed.tech/tags/inference.md>), [language](<https://devfeed.tech/tags/language.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [latency](<https://devfeed.tech/tags/latency.md>), [model](<https://devfeed.tech/tags/model.md>), [performance](<https://devfeed.tech/tags/performance.md>), [post-training](<https://devfeed.tech/tags/post-training.md>), [qwen](<https://devfeed.tech/tags/qwen.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [reasoning-model](<https://devfeed.tech/tags/reasoning-model.md>)

### AI overview

GLiDE transfers reasoning capability into single-pass decision models that score candidate answers without generating text or intermediate reasoning. The 0.4B-27B family uses Qwen hybrid language models with bidirectional interactions among context, question, and candidates, plus soft-target distillation from multiple teachers--including a reasoning model--and progressive post-training on varied decision tasks. Parameter-free recurrent depth adds inference-time computation without token generation. The abstract reports that each of five sizes outperforms evaluated alternatives of equal or smaller size on the Hugging Face benchmark. The 27B model scores 71.52 on a hidden set with 2.0% expected calibration error and 92.9 ms median latency on one GPU; checkpoint averaging and recurrent depth raise the score to 73.37, with 2.1% calibration error.

### Source excerpt

arXiv:2611.26509v1 Announce Type: new Abstract: Decision models enable fast, confidence-aware decisions by scoring candidate answers in a single forward pass, without generating text or intermediate reasoning. Yet reasoning can substantially improve decision quality, raising the question of whether its benefits can be retained without the cost of generating it. We investigate this question with \textbf{GLiDE}, a family of decision models spanning 0.4B to 27B parameters. Building on Qwen hybrid language models, GLiDE enables bidirectional interactions between the context, question, and candidate answers by removing causal masking from full-attention layers. To transfer reasoning capabilities into single-pass inference, we combine soft-target distillation from multiple teachers, including a reasoning model, with progressive post-training on diverse decision tasks. We further explore parameter-free recurrent depth to increase inference-time computation without generating tokens. Across five model sizes, GLiDE lies on the performance--parameter Pareto frontier of the Hugging Face \DI{}, with each model outperforming evaluated alternatives of equal or smaller size. \dmtwo{} (27B) achieves a hidden-set score of 71.52, with an expected calibration error of 2.0\% and a median latency of 92.9\,ms on a single GPU. Further checkpoint averaging and parameter-free recurrent depth yield GLiDE~27B, which reaches 73.37 with an expected calibration error of 2.1\%.

## The Coverage Selection Gap: Ranking, Not Generation, Limits LLM Rubric Construction

DevFeed: [The Coverage Selection Gap: Ranking, Not Generation, Limits LLM Rubric Construction](<https://devfeed.tech/articles/the-coverage-selection-gap-ranking-not-generation-limits-llm-rubric-construction-145536.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25557>)

Author: Ali Karkehabadi, Jamshid Hassanpour, Rafael E. Banchs

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>)

Tags: [1-5](<https://devfeed.tech/tags/1-5.md>), [ablation](<https://devfeed.tech/tags/ablation.md>), [acquisitions](<https://devfeed.tech/tags/acquisitions.md>), [automatic](<https://devfeed.tech/tags/automatic.md>), [axis](<https://devfeed.tech/tags/axis.md>), [bottleneck](<https://devfeed.tech/tags/bottleneck.md>), [budget](<https://devfeed.tech/tags/budget.md>), [candidate](<https://devfeed.tech/tags/candidate.md>), [construction](<https://devfeed.tech/tags/construction.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [diagnostic](<https://devfeed.tech/tags/diagnostic.md>), [difference](<https://devfeed.tech/tags/difference.md>), [discovery](<https://devfeed.tech/tags/discovery.md>), [effective](<https://devfeed.tech/tags/effective.md>), [gap](<https://devfeed.tech/tags/gap.md>), [generation](<https://devfeed.tech/tags/generation.md>), [limits](<https://devfeed.tech/tags/limits.md>), [llm](<https://devfeed.tech/tags/llm.md>), [ranking](<https://devfeed.tech/tags/ranking.md>), [selection](<https://devfeed.tech/tags/selection.md>)

### AI overview

In M&A due diligence rubric construction, broadening how a system searches source evidence improved criterion recall more than generating additional candidates along existing search directions. The multi-axis framework combines structural and free-form views to build a candidate pool, then measures the coverage-selection gap between reference-free ranking and a reference-informed greedy diagnostic under a fixed selection budget. Across 16 development tasks and four held-out tasks, it reached 0.876 macro recall against senior-authored rubrics using an LLM-based equivalence protocol. Ablations found new structural search axes raised recall by 6-10 percentage points, while more generation along existing directions added at most 1.5 points. The results indicate that selecting from a high-coverage pool remains a substantial bottleneck; the evidence is limited to the described M&A setting and evaluation protocol.

### Source excerpt

arXiv:2611.25557v1 Announce Type: cross Abstract: Task-specific rubrics are increasingly used to evaluate open-ended language-model outputs, yet constructing comprehensive evaluation criteria from complex source documents remains challenging. We study automatic rubric construction in professional mergers and acquisitions (M&A) due diligence, where evaluation criteria must be inferred from task instructions and supporting documents without access to an expert-authored reference rubric. We introduce a multi-axis generation framework that explores complementary structural and free-form views of the source evidence to construct a broad pool of candidate evaluation criteria. To distinguish failures of criterion discovery from failures of candidate selection, we introduce the coverage-selection gap, which measures the difference between reference-free ranking and a reference-informed greedy diagnostic under a fixed selection budget. Developed on 16 tasks and evaluated on four held-out tasks, our framework achieves 0.876 macro recall against senior-authored reference rubrics under an LLM-based criterion-equivalence protocol. Ablation studies further show that introducing new structural search axes improves recall by 6-10 percentage points, whereas increasing generation volume along existing search directions yields at most 1.5 points. These findings suggest that, once a high-coverage candidate pool has been constructed, effective reference-free selection becomes a substantial bottleneck in document-grounded rubric construction.

## Forget Freely, Grep Later: Lossless, Index-Free Memory for Long-Horizon LLM Agents

DevFeed: [Forget Freely, Grep Later: Lossless, Index-Free Memory for Long-Horizon LLM Agents](<https://devfeed.tech/articles/forget-freely-grep-later-lossless-index-free-memory-for-long-horizon-llm-agents-145470.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25123>)

Author: Ziyan Yang, Taiwei Shi, Jieyu Zhao

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>)

Tags: [agent-memory](<https://devfeed.tech/tags/agent-memory.md>), [agents](<https://devfeed.tech/tags/agents.md>), [context-management](<https://devfeed.tech/tags/context-management.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [embeddings](<https://devfeed.tech/tags/embeddings.md>), [expression](<https://devfeed.tech/tags/expression.md>), [grep](<https://devfeed.tech/tags/grep.md>), [index](<https://devfeed.tech/tags/index.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-agents](<https://devfeed.tech/tags/llm-agents.md>), [log](<https://devfeed.tech/tags/log.md>), [memory](<https://devfeed.tech/tags/memory.md>), [qwen3-5](<https://devfeed.tech/tags/qwen3-5.md>), [recall](<https://devfeed.tech/tags/recall.md>), [regular](<https://devfeed.tech/tags/regular.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [research](<https://devfeed.tech/tags/research.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [search](<https://devfeed.tech/tags/search.md>), [windows](<https://devfeed.tech/tags/windows.md>)

### AI overview

GrepMem stores an agent's complete history in an append-only plain-text log and uses regular-expression search for on-demand recall. This avoids the embeddings, retrieval index, memory extraction, and additional models used by structured memory approaches, while sidestepping losses from summarization or masking. In evaluations with Qwen3.5-9B trained by reinforcement learning, GrepMem reached 95-97% accuracy on ContextBench, compared with at most 21% for lossy baselines, and exceeded SUPO's summarization by 8.8 percentage points on BCP-Multi. The authors report effective scaling to longer streams and a position on the accuracy-token-cost Pareto frontier across five tasks. These results are limited to the reported evaluations; the supplied excerpt does not describe implementation details or broader deployment evidence.

### Source excerpt

arXiv:2611.25123v1 Announce Type: cross Abstract: Long-horizon LLM agents often exceed their context windows, forcing them to discard information that may be needed later. Existing context management strategies rely on summarization, masking, or structured memory, requiring decisions about what to preserve or how to retrieve it before future needs are known. However, these methods can introduce lossy compression, potentially discarding details needed for future tasks. To address this limitation, we introduce GrepMem, a simple, lossless, and index-free memory mechanism built on two components: an append-only plain-text log of the agent's complete history and a regular-expression search tool for on-demand recall. Unlike structured memory systems, GrepMem requires no embeddings, retrieval index, memory extraction, or additional models. We train Qwen3.5-9B with reinforcement learning under five context management strategies and evaluate on multi-question research sessions, long-stream recall, and agent-trajectory question answering. GrepMem achieves 95-97% accuracy on ContextBench, versus at most 21% for lossy baselines, and outperforms SUPO's summarization by 8.8 percentage points on BCP-Multi. GrepMem scales effectively to increasingly long input streams while remaining on the accuracy-token-cost Pareto frontier across all five evaluation tasks. Our results show that a plain-text log and grep provide a simple but competitive approach to long-horizon agent memory, without requiring complex structured memory infrastructure.

## Reasoning-Trace Analysis Using Large Language Models for Interpretable Q-Matrix Construction

DevFeed: [Reasoning-Trace Analysis Using Large Language Models for Interpretable Q-Matrix Construction](<https://devfeed.tech/articles/reasoning-trace-analysis-using-large-language-models-for-interpretable-q-matrix-construction-145670.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26422>)

Author: Jihong Zhang, Zijing Jin, Mingfeng Xue

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Chain-of-thought](<https://devfeed.tech/topics/chain-of-thought.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [chain-of-thought](<https://devfeed.tech/tags/chain-of-thought.md>), [construction](<https://devfeed.tech/tags/construction.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [framework](<https://devfeed.tech/tags/framework.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [overlap](<https://devfeed.tech/tags/overlap.md>), [rta](<https://devfeed.tech/tags/rta.md>), [simulation](<https://devfeed.tech/tags/simulation.md>), [study](<https://devfeed.tech/tags/study.md>), [traces](<https://devfeed.tech/tags/traces.md>)

### AI overview

Reasoning-Trace Analysis (RTA) makes individual LLM-generated Q-matrix assignments inspectable by having models produce attribute-labeled reasoning steps, then using those traces to reconstruct and compare assignments. In simulations involving perturbed fraction-subtraction items, four LLMs identified required attributes and omitted unnecessary ones. In an empirical study of 11 items and 536 examinees, CoT-derived matrices had substantial but lower mean overlap with the reference matrix than directly generated matrices (.809 versus .877), and varied more across replications. Trace inspection connected disagreements to item-type coding, labeling rules, and how operations were represented as steps; matrices with similar overlap could still differ in fit to responses. The authors position RTA as supplementary evidence for expert review and empirical validation.

### Source excerpt

arXiv:2611.26422v1 Announce Type: new Abstract: Large language models (LLMs) can generate Q-matrices for diagnostic classification models that approximate expert specifications, but evaluations have focused on the final matrices, leaving the basis of individual item-attribute assignments unexamined. This opacity limits the responsible use of AI in measurement. We propose Reasoning-Trace Analysis (RTA), a framework in which an LLM generates attribute-labeled reasoning steps through chain-of-thought (CoT) prompting and these steps are analyzed to reconstruct, compare, and inspect each assignment. RTA is intended to supplement expert review and empirical validation. In a simulation study with perturbed fraction subtraction items, four LLMs recovered the required attributes and withheld attributes the items did not require. In an empirical study with 11 fraction subtraction items, responses from 536 examinees, four LLMs, up to three temperatures, and 100 replications per condition, CoT-derived Q-matrices overlapped substantially with the reference Q-matrix (mean overlap .809) but less than directly generated Q-matrices (.877), and they varied more across replications. Inspection of the traces linked disagreements to item-type coding, labeling rules, and how operations were expressed as steps. Q-matrices with similar overlap differed in fit to examinee responses. RTA provides inspectable evidence that matrix-level agreement and fit do not.

[Next page](<https://devfeed.tech/topics/llm-reasoning.md?cursor=WyIyMDI2LTExLTIwVDA1OjAwOjAwKzAwOjAwIiwgIjkwMTdmOTA5LTQ3MTQtNGFmMC04NjlhLTAyYWZiOWE3NDc3NyJd>)