# vlm

A vision-language model that processes image and text inputs to generate text outputs.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## When Safety Rationales Hallucinate: Diagnosing and Grounding Multimodal Safety Guards

DevFeed: [When Safety Rationales Hallucinate: Diagnosing and Grounding Multimodal Safety Guards](<https://devfeed.tech/articles/when-safety-rationales-hallucinate-diagnosing-and-grounding-multimodal-safety-guards-145588.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25793>)

Author: Yingshuo Wang, Weixiang Zhao, Guancheng Li, Jiahe Guo, Yu Zhang, Yanyan Zhao

Published: 2026-11-20T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [vlm](<https://devfeed.tech/topics/vlm.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>)

Tags: [analysis](<https://devfeed.tech/tags/analysis.md>), [attribution](<https://devfeed.tech/tags/attribution.md>), [classification](<https://devfeed.tech/tags/classification.md>), [classification-accuracy](<https://devfeed.tech/tags/classification-accuracy.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [fine](<https://devfeed.tech/tags/fine.md>), [focus](<https://devfeed.tech/tags/focus.md>), [generation](<https://devfeed.tech/tags/generation.md>), [grounded](<https://devfeed.tech/tags/grounded.md>), [grounding](<https://devfeed.tech/tags/grounding.md>), [guards](<https://devfeed.tech/tags/guards.md>), [learning](<https://devfeed.tech/tags/learning.md>), [model](<https://devfeed.tech/tags/model.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [safety](<https://devfeed.tech/tags/safety.md>), [scale](<https://devfeed.tech/tags/scale.md>)

### AI overview

The study finds that multimodal safety guards often produce unreliable rationales: they may state unsupported visual facts or assign incorrect safety meanings to grounded content. This persists across the examined methods, vision-language backbones, and datasets, and does not consistently improve with higher prediction accuracy or model scale. In Think-with-Image analysis, errors cluster around fine-grained attributes and relations; visual tools are often used infrequently or without enough spatial focus. The proposed Tool-Contrastive Grounding Reinforcement Learning (TCG-RL) rewards tool use only when it improves grounding relative to a tool-free reference. The abstract reports substantially fewer rationale hallucinations while maintaining competitive safety prediction performance, highlighting that prediction correctness alone does not establish rationale reliability.

### Source excerpt

arXiv:2611.25793v1 Announce Type: new Abstract: Multimodal safety guards increasingly generate rationales alongside safety predictions, yet whether these rationales are reliable remains largely unexplored. We systematically study *rationale hallucination* from two perspectives: visual rationale hallucination, where rationales contain unsupported visual facts, and safety attribution hallucination, where grounded content is assigned an incorrect safety interpretation. Across rationale-generation methods, multiple vision-language backbones, and safety datasets, we find that rationale hallucination is widespread and does not consistently diminish with higher classification accuracy or increased model scale. Further analysis of the Think-with-Image paradigm shows that hallucination is concentrated in fine-grained attributes and relations, while visual tools are often invoked infrequently or without sufficient spatial focus. Based on these findings, we propose *Tool-Contrastive Grounding Reinforcement Learning* (TCG-RL), which rewards tool use only when it improves rationale grounding over a tool-free reference. TCG-RL substantially reduces rationale hallucination while maintaining competitive safety prediction performance. By revealing the gap between prediction correctness and rationale reliability, our study motivates a shift in multimodal safety guarding from outcome-only evaluation toward safety decisions that are both accurate and verifiably grounded.

## Bridging vision-language reasoning and learned control for legged navigation in the wild

DevFeed: [Bridging vision-language reasoning and learned control for legged navigation in the wild](<https://devfeed.tech/articles/bridging-vision-language-reasoning-and-learned-control-for-legged-navigation-in-the-wild-145484.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25215>)

Author: Takahiro Miki, Robin Schmid, Manthan Patel, Turcan Tuna, Julia Richter, Julian Nubert, Joonho Lee, Matthias M\\"uller, Marco Hutter

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Robot Navigation](<https://devfeed.tech/topics/robot-navigation.md>), [Robotics](<https://devfeed.tech/topics/robotics.md>), [vlm](<https://devfeed.tech/topics/vlm.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [autonomous-navigation](<https://devfeed.tech/tags/autonomous-navigation.md>), [control](<https://devfeed.tech/tags/control.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-ro](<https://devfeed.tech/tags/cs-ro.md>), [legged-robots](<https://devfeed.tech/tags/legged-robots.md>), [navigation](<https://devfeed.tech/tags/navigation.md>), [planning](<https://devfeed.tech/tags/planning.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [real-world](<https://devfeed.tech/tags/real-world.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [risk](<https://devfeed.tech/tags/risk.md>), [robotics](<https://devfeed.tech/tags/robotics.md>), [spatial-reasoning](<https://devfeed.tech/tags/spatial-reasoning.md>), [vision](<https://devfeed.tech/tags/vision.md>)

### AI overview

ROAM combines a vision-language model (VLM) with two reinforcement-learning policies to help legged robots navigate complex alpine and urban environments. The VLM interprets scenes, assesses risks, and assigns semantic navigation costs; open-vocabulary segmentation turns these labels into a real-time risk map that conditions the policies for locomotion and local navigation. Reported real-world trials include kilometer-long hikes with minimal intervention, interpreting signs to avoid danger, and identifying safer places to step. The abstract also flags VLM reliability and spatial reasoning as unresolved challenges, so foundational models are useful components but not a complete solution for navigation in the wild.

### Source excerpt

arXiv:2611.25215v1 Announce Type: cross Abstract: Autonomous navigation in diverse real-world environments remains one of the greatest challenges in robotics. It demands both robust low-level motion control to handle diverse terrain and high-level reasoning to interpret environmental cues. We present ROAM (Robust Open-world Autonomous Mobility), a system that enables legged robots to perform kilometer-scale navigation in complex alpine and urban settings. ROAM combines the world knowledge of a Vision-Language Model (VLM), pre-trained on internet-scale data, with robust locomotion and local planning policies learned in simulation via reinforcement learning (RL). The VLM provides common-sense reasoning and semantic interpretation of the scene, assessing risks, interpreting signs, and producing semantic labels with associated navigation costs. An open-vocabulary segmentation module converts these labels into a dense, real-time risk map, which conditions two pre-trained RL policies responsible for agile locomotion and local navigation, effectively coupling high-level VLM reasoning with low-level physical control. Real-world experiments showed the system's ability to autonomously navigate a wide range of complex real-world environments. The robot completes kilometer-long hikes with minimal intervention, interprets signs to avoid danger, and evaluates its environment to identify safe areas on which to step. This work represents a step toward the reliable deployment of systems in unstructured environments. However, it also underscores the challenges that remain in VLM reliability and spatial reasoning, indicating that foundational models are a crucial component but not yet a complete solution for navigation in the wild.

## Relax Within, Balance Across: Load Balancing under Composition Shifts in Vision-Language Mixture-of-Experts

DevFeed: [Relax Within, Balance Across: Load Balancing under Composition Shifts in Vision-Language Mixture-of-Experts](<https://devfeed.tech/articles/relax-within-balance-across-load-balancing-under-composition-shifts-in-vision-language-mixture-of-experts-145359.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2608.00574>)

Author: Ziang Wu, Peng Jin, Qishen Yin, Munan Ning, Hao Li, Peizhen Zhang, Huihui Zhou, Li Yuan

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [vlm](<https://devfeed.tech/topics/vlm.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>)

Tags: [balance](<https://devfeed.tech/tags/balance.md>), [balancing](<https://devfeed.tech/tags/balancing.md>), [cancellation](<https://devfeed.tech/tags/cancellation.md>), [changes](<https://devfeed.tech/tags/changes.md>), [code](<https://devfeed.tech/tags/code.md>), [composition](<https://devfeed.tech/tags/composition.md>), [counts](<https://devfeed.tech/tags/counts.md>), [cross](<https://devfeed.tech/tags/cross.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [errors](<https://devfeed.tech/tags/errors.md>), [experts](<https://devfeed.tech/tags/experts.md>), [fast](<https://devfeed.tech/tags/fast.md>), [fixed](<https://devfeed.tech/tags/fixed.md>), [flat](<https://devfeed.tech/tags/flat.md>), [gap](<https://devfeed.tech/tags/gap.md>), [image](<https://devfeed.tech/tags/image.md>), [language](<https://devfeed.tech/tags/language.md>), [load](<https://devfeed.tech/tags/load.md>), [load-balancing](<https://devfeed.tech/tags/load-balancing.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [vision](<https://devfeed.tech/tags/vision.md>)

### AI overview

On a fixed vision-language MoE router, balancing image and text tokens in one pooled auxiliary loss can conceal opposing modality load errors: they cancel near the training image-to-text ratio, but load skews as resolution, tiling, or image count changes that mix. The paper derives a quadratic response law under fixed modality profiles, with the image-text load gap governing sensitivity. It also reports that patches from an image route similarly as patch counts vary, allowing large images to dominate flat pooling. ReBA separates modality losses by token fraction and weights images equally, allowing per-image specialization while balancing visual load overall. Across four split backbones, the authors report stable balance across resolutions, tilings, and image counts without hurting accuracy; large modality gaps in released native MoEs suggest potential for native post-training.

### Source excerpt

arXiv:2608.00574v2 Announce Type: replace-cross Abstract: Vision-language mixture-of-experts (MoE) models such as Qwen3-VL need balanced loads for fast parallel serving. Standard training pools image and text tokens into one auxiliary loss, assuming training balance holds at inference. Yet we find a surprising mismatch: on a fixed router, balance shifts sharply whenever resolution, dynamic tiling, or image count changes the image-to-text token ratio. Splitting load by modality reveals cross-modal load cancellation: opposing image and text errors cancel near the training ratio, hiding strong skew until the mix changes. To quantify this shift, we derive an exact quadratic law: under fixed modality profiles, one reference run predicts the full response curve, and the image-text gap governs sensitivity. Meanwhile, patches within an image route similarly while patch counts vary, letting large images dominate flat pooling. We propose ReBA (Relax Within, Balance Across) to separate modality losses by token fraction and weight images equally, letting each image specialize while keeping overall visual load even. Across four split backbones, ReBA turns fragile single-ratio balance into stable balance across resolutions, tilings, and image counts without hurting accuracy. Large modality gaps in released native MoEs also show its promise for native post-training. Code is available at https://github.com/ZiangWu-77/ReBA.

## SKILL.PNG: Learning Agent Skills in Pixel Space

DevFeed: [SKILL.PNG: Learning Agent Skills in Pixel Space](<https://devfeed.tech/articles/skill-png-learning-agent-skills-in-pixel-space-145160.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26296>)

Author: Maximilian Seeth, Chengzhi Martin Hu, Marie Kempkes, Paul R\\"ottger, Leo Schwinn

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Agent Skills](<https://devfeed.tech/topics/agent-skills.md>), [vlm](<https://devfeed.tech/topics/vlm.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Model Development](<https://devfeed.tech/topics/model-development.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [gradient-descent](<https://devfeed.tech/tags/gradient-descent.md>), [hybrid](<https://devfeed.tech/tags/hybrid.md>), [image](<https://devfeed.tech/tags/image.md>), [instructions](<https://devfeed.tech/tags/instructions.md>), [language](<https://devfeed.tech/tags/language.md>), [learning](<https://devfeed.tech/tags/learning.md>), [llms](<https://devfeed.tech/tags/llms.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [optimized](<https://devfeed.tech/tags/optimized.md>), [performance](<https://devfeed.tech/tags/performance.md>), [pixel](<https://devfeed.tech/tags/pixel.md>), [pixels](<https://devfeed.tech/tags/pixels.md>), [skill](<https://devfeed.tech/tags/skill.md>), [vision-language-models](<https://devfeed.tech/tags/vision-language-models.md>), [vision-language-models-vlms](<https://devfeed.tech/tags/vision-language-models-vlms.md>), [vlms](<https://devfeed.tech/tags/vlms.md>)

### AI overview

Skill.png represents agent skills as images whose pixels are optimized by gradient descent through the vision encoder of a frozen vision-language model (VLM). The authors report that these image skills beat frontier-LLM-optimized textual skills in 9 of 16 settings across four VLMs and four benchmarks, using 64-256 image tokens at inference versus about 900-2,000 text tokens. Jointly optimized text and image skills further improve performance. An image optimized across an ensemble can transfer among its VLMs, while the reported compatibility claim is for VLMs. The abstract describes the evaluation results but does not provide per-setting details.

### Source excerpt

arXiv:2611.26296v1 Announce Type: new Abstract: AI agents are commonly specialized for new tasks and domains using skills, i.e. natural language instructions placed in their context. However, the discrete nature of textual skills prevents continuous optimization. Gradient-based approaches such as finetuning, on the other hand, are typically model-specific, preventing skill transfer across different agents. Since many agents today are built on vision-language models (VLMs), we can address these limitations by introducing Skill.png: a method for obtaining skill-like image inputs whose pixels are optimized with gradient descent through the vision encoder of a frozen VLM. Image skills are compatible with any VLM, and a single image can be optimized jointly for an ensemble of models. We show that Skill.png significantly outperforms state-of-the-art textual skills optimized using frontier LLMs in 9 of 16 settings (across 4 VLMs and 4 benchmarks) while requiring only 64 to 256 image tokens instead of roughly 900 to 2000 text tokens at inference time. Hybrid skills, i.e. text and image skills jointly optimized, further improve agent performance. Moreover, an image optimized with an ensemble of VLMs can successfully transfer across the models in the ensemble. Skill.png thus presents the first agent skill that can be optimized with gradient descent while retaining transfer benefits of text skills. Our code is available at: https://github.com/omseeth/skill.png

## Probing the "Conscious" Mind: Selective Access and Global Workspace in Vision-Language Models

DevFeed: [Probing the "Conscious" Mind: Selective Access and Global Workspace in Vision-Language Models](<https://devfeed.tech/articles/probing-the-conscious-mind-selective-access-and-global-workspace-in-vision-language-models-145196.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26590>)

Author: Jiahao Huang, Jun Wang

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [vlm](<https://devfeed.tech/topics/vlm.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>)

Tags: [accessibility](<https://devfeed.tech/tags/accessibility.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decoder](<https://devfeed.tech/tags/decoder.md>), [encoder](<https://devfeed.tech/tags/encoder.md>), [jacobian](<https://devfeed.tech/tags/jacobian.md>), [lens](<https://devfeed.tech/tags/lens.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [tasks](<https://devfeed.tech/tags/tasks.md>), [theory](<https://devfeed.tech/tags/theory.md>), [training](<https://devfeed.tech/tags/training.md>), [vision-language-models](<https://devfeed.tech/tags/vision-language-models.md>)

### AI overview

MC-Lens maps intermediate vision-encoder and language-decoder states into a shared semantic space based on the decoder's output vocabulary, enabling comparison of concept accessibility across modalities and interventions on accessed concepts. The authors report that concepts are readable in intermediate states, selectively accessed depending on the task, and involved in downstream reasoning in counterfactual experiments. Their two J-space metrics track cross-modal alignment during controlled training more clearly than hidden-state cosine; reliable selective access develops later than concept accessibility. Across VLM families and sizes, the abstract reports similar accessibility, query-dependent selectivity, and cross-task intervention patterns despite different internal organization. The evidence provided is an abstract-level excerpt, so it does not detail methods, quantitative results, or limitations beyond these claims.

### Source excerpt

arXiv:2611.26590v1 Announce Type: cross Abstract: Vision-language models (VLMs) integrate visual and textual information, but their outputs do not reveal internal concept representations. Global workspace theory (GWT) proposes that information is selected from parallel brain processes, integrated in a shared workspace, and made available to specialised processes. We introduce the Multimodal Concept Lens (MC-Lens), a multimodal Jacobian lens, to study these functions in VLM concept processing. Its vision encoder and language decoder lenses map intermediate states to a shared semantic space defined by the decoder's output vocabulary to compare concept accessibility across modalities. MC-Lens supports internal concept interventions to examine the role of accessed concepts in downstream computation. (i) Accessibility and selective access. Multimodal concept information is readable in intermediate states and selectively accessible as tasks require. We extend "readout carriers" from one name token to a token set to better capture and express concept information and support counterfactual experiments. (ii) Concepts in downstream computation. Counterfactual experiments show that accessed concept information participates in downstream computation and reasoning. (iii) Development of cross-modal alignment and selective access. We develop two J-space metrics tracking the development of cross-modal concept alignment more clearly than hidden state cosine in controlled VLM training. The experiment also shows that reliable selective access develops later than concept accessibility. (iv) Shared functions across models. Across VLM families and sizes, we find shared patterns of concept accessibility, query-dependent selectivity and cross-task intervention effects despite differences in internal organisation. These findings offer an experimental basis to study GWT in multimodal artificial systems and how it can guide cross-modal alignment improvements.

## From Portraits to Pathology: Probing Diagnostic Overreach in Vision-Language Models

DevFeed: [From Portraits to Pathology: Probing Diagnostic Overreach in Vision-Language Models](<https://devfeed.tech/articles/from-portraits-to-pathology-probing-diagnostic-overreach-in-vision-language-models-145368.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.05540>)

Author: Karan Dua, Amit Agarwal, Hitesh Laxmichand Patel, Hansa Meghwani, Jyotika Singh, Ranjeet Gupta, Graham Horwood, Tao Sheng, Avi Sil, Sujith Ravi, Dan Roth

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [vlm](<https://devfeed.tech/topics/vlm.md>), [Computer vision](<https://devfeed.tech/topics/computer-vision.md>)

Tags: [adhd](<https://devfeed.tech/tags/adhd.md>), [anxiety](<https://devfeed.tech/tags/anxiety.md>), [appearance](<https://devfeed.tech/tags/appearance.md>), [computer-vision](<https://devfeed.tech/tags/computer-vision.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [diagnostic](<https://devfeed.tech/tags/diagnostic.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [guardrail](<https://devfeed.tech/tags/guardrail.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [parity](<https://devfeed.tech/tags/parity.md>), [safety](<https://devfeed.tech/tags/safety.md>), [semantic](<https://devfeed.tech/tags/semantic.md>), [vision-language-models](<https://devfeed.tech/tags/vision-language-models.md>), [vision-language-models-vlms](<https://devfeed.tech/tags/vision-language-models-vlms.md>)

### AI overview

The study finds that vision-language models can infer ASD, ADHD, major depressive disorder, and generalized anxiety disorder from facial expressions, despite appearance alone being insufficient for diagnosis. On PARITY, a demographically balanced dataset designed to vary expressions while keeping identity and other visual factors as constant as possible, 27 open- and closed-weight models showed inconsistent abstention across expressions, conditions, and task formats. The problem was widespread among open-weight models and also appeared in safety-trained frontier models that abstained on single portraits but made judgments when comparing portraits of the same person. Clinical instructions did not reliably prevent such judgments, and a guardrail model rarely flagged paired requests. Most baseline open-model refusals also failed to explain both why facial evidence is insufficient and why clinical assessment is needed. The authors warn that unsupported judgments and authoritative-sounding explanations may cause distress, reinforce stigma, or offer false reassurance.

### Source excerpt

arXiv:2609.05540v3 Announce Type: replace-cross Abstract: Many medical conditions require clinical assessment of symptoms, history, and functioning rather than judgments from appearance. We study whether vision-language models (VLMs) abstain from facial attribution of autism spectrum disorder (ASD), attention-deficit/hyperactivity disorder (ADHD), major depressive disorder (MDD), and generalized anxiety disorder (GAD). To support this study, we construct PARITY (Paired Assessment with Reused Identity), a demographically balanced dataset for testing whether changing facial expressions alters diagnostic judgments, while holding identity and other visual factors as constant as possible. Our comprehensive evaluation of 27 open- and closed-weight models reveals inconsistent abstention across expressions, conditions, and task formats. Models turn facial expressions into unsupported medical judgments: this behavior is widespread among open-weight VLMs and extends to safety-trained frontier models that abstain on individual portraits yet make diagnostic judgments when comparing portraits of the same identity. Safeguard checks reveal further gaps: clinical instructions do not consistently prevent these judgments, and a guardrail model rarely flags paired requests as unsafe. Semantic analysis further shows that most baseline open-model refusals fail to explain both the insufficiency of facial evidence and the need for clinical assessment. Unsupported judgments and seemingly authoritative explanations can cause distress, reinforce stigma, and offer false reassurance, underscoring the need for stronger safety measures in model training and deployment.

## HiUni: Harmonizing Understanding and Generation in Shared Visual Space

DevFeed: [HiUni: Harmonizing Understanding and Generation in Shared Visual Space](<https://devfeed.tech/articles/hiuni-harmonizing-understanding-and-generation-in-shared-visual-space-145631.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26107>)

Author: Jiahao Hu, Jingdong Du, Yichen Zhang, Yi Song, Zhenyu Zhang, Jiaxin Niu, Ruihang Li, Xiangzhao Hao, Yiqian Zhang, Yong Huang, Zefeng Zhang, Zheyuan Gu, Shuohuan Wang

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [diffusion-transformers](<https://devfeed.tech/topics/diffusion-transformers.md>), [vlm](<https://devfeed.tech/topics/vlm.md>), [Image](<https://devfeed.tech/topics/image.md>)

Tags: [comparisons](<https://devfeed.tech/tags/comparisons.md>), [computer-vision](<https://devfeed.tech/tags/computer-vision.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [generation](<https://devfeed.tech/tags/generation.md>), [image-generation](<https://devfeed.tech/tags/image-generation.md>), [key](<https://devfeed.tech/tags/key.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [pixel](<https://devfeed.tech/tags/pixel.md>), [semantic](<https://devfeed.tech/tags/semantic.md>), [sharing](<https://devfeed.tech/tags/sharing.md>), [spatial](<https://devfeed.tech/tags/spatial.md>), [synthesis](<https://devfeed.tech/tags/synthesis.md>), [tasks](<https://devfeed.tech/tags/tasks.md>), [transformer](<https://devfeed.tech/tags/transformer.md>), [vision-language-model](<https://devfeed.tech/tags/vision-language-model.md>), [vit](<https://devfeed.tech/tags/vit.md>)

### AI overview

HiUni extends a pretrained vision-language model for understanding and image generation using the same language-aligned ViT and dense Transformer backbone, with task-specific computation confined to the output. To address the difficulty of optimizing pixel-space generation and the loss of spatial detail in the pretrained ViT, it adds a Pixel Diffusion Head and a lightweight HiGate pathway. The abstract reports an average understanding score similar to the pretrained model and competitive image generation and editing performance; it provides no specific scores or benchmark details.

### Source excerpt

arXiv:2611.26107v1 Announce Type: cross Abstract: Unified multimodal models aim to support visual understanding and generation within a single framework. However, prevailing approaches often separate their visual representations or backbone computation to accommodate the different requirements of these two tasks, which limits the degree of unification. In this paper, we present HiUni, which revisits this design choice and explores whether a pretrained vision language model can be directly extended into a unified model with extensive parameter sharing. Specifically, HiUni reuses the same language-aligned ViT and dense Transformer backbone for both capabilities, while confining task-specific computation to the output side. Yet such a design introduces two key challenges. Generation in pixel space is difficult to optimize, and semantic abstraction in pretrained ViT discards fine spatial details required for synthesis. To address these challenges, HiUni introduces a \emph{Pixel Diffusion Head} tailored to direct pixel-space optimization and a lightweight \emph{HiGate} pathway that provides fine-grained spatial details near the output stage. Extensive experiments show that HiUni achieves a similar average understanding score to the pretrained VLM while attaining competitive performance on image generation and editing. Controlled comparisons and analyses further demonstrate that a single dense backbone can effectively support both capabilities without task-specific backbone computation, highlighting a simpler path toward unified multimodal modeling.

## CoachSI: Self-Improvement of Robot Policies with a Language Coach

DevFeed: [CoachSI: Self-Improvement of Robot Policies with a Language Coach](<https://devfeed.tech/articles/coachsi-self-improvement-of-robot-policies-with-a-language-coach-145529.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25529>)

Author: Jiachen Tao, Gengyu Zhang, Haoxuan Wang, Bing Liu, Yan Yan

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Robot Navigation](<https://devfeed.tech/topics/robot-navigation.md>), [vlm](<https://devfeed.tech/topics/vlm.md>), [post-training](<https://devfeed.tech/topics/post-training.md>)

Tags: [camera](<https://devfeed.tech/tags/camera.md>), [coaching](<https://devfeed.tech/tags/coaching.md>), [completion](<https://devfeed.tech/tags/completion.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-ro](<https://devfeed.tech/tags/cs-ro.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [horizon](<https://devfeed.tech/tags/horizon.md>), [improvement](<https://devfeed.tech/tags/improvement.md>), [instructions](<https://devfeed.tech/tags/instructions.md>), [language](<https://devfeed.tech/tags/language.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [model](<https://devfeed.tech/tags/model.md>), [naming](<https://devfeed.tech/tags/naming.md>), [next](<https://devfeed.tech/tags/next.md>), [policies](<https://devfeed.tech/tags/policies.md>), [policy](<https://devfeed.tech/tags/policy.md>), [robot](<https://devfeed.tech/tags/robot.md>), [self-improvement](<https://devfeed.tech/tags/self-improvement.md>), [vision](<https://devfeed.tech/tags/vision.md>), [vision-language-action](<https://devfeed.tech/tags/vision-language-action.md>), [vision-language-model](<https://devfeed.tech/tags/vision-language-model.md>), [vlm](<https://devfeed.tech/tags/vlm.md>)

### AI overview

CoachSI improves long-horizon robot task performance by having a frozen vision-language model watch the robot and name the next step in the policy's atomic-instruction format; it never supplies actions. Successful coached runs are relabeled with the original task instruction and used to fine-tune the policy, which is then deployed without a coach. The action backbone remains frozen. On stalled states, next-step coaching raises subgoal completion 4.6-fold over the unchanged task instruction. Unassisted success in unseen scenes rises from 47.4% to 59.3%, exceeding human-demonstration fine-tuning by 9.8 percentage points and equal-budget self-collected-success fine-tuning by 7.3 points. With a coach limited to camera images, success reaches 56.8%, above both baselines.

### Source excerpt

arXiv:2611.25529v1 Announce Type: cross Abstract: Generalist robot policies execute atomic skills reliably but fail when the same skills are composed into long-horizon tasks. Self-improvement from the policy's own experience does not close this gap: experience collected by the policy alone stays within its own distribution and reproduces what it can already do. We show that what the policy lacks is not a skill but the next step. We present CoachSI, a self-improvement method in which a frozen, off-the-shelf vision-language model (VLM) coaches the policy in language: it watches the robot and names the next step in the format of the policy's atomic instructions, and never outputs an action. Successful coached rollouts are relabeled in hindsight with the original task instruction and used to fine-tune the policy. The fine-tuned policy is then deployed on its own. On states where a World Action Model and a Vision-Language-Action model have stalled, naming the next step raises subgoal completion by 4.6x over the unchanged task instruction, and the frozen VLM supplies it from the same camera images the policy sees. After fine-tuning, unassisted success on long-horizon tasks in unseen scenes rises from 47.4% to 59.3%, 9.8 points above fine-tuning on human demonstrations and 7.3 points above fine-tuning on self-collected successes at an equal data budget. With a coach that sees only camera images, it reaches 56.8%, above both baselines. The gain is obtained with the action backbone frozen: language coaching teaches the policy what should happen next, and the skills it already has do the rest.

## ZeroVLN: Learning to Navigate without Supervised Fine-Tuning of the VLM Brain

DevFeed: [ZeroVLN: Learning to Navigate without Supervised Fine-Tuning of the VLM Brain](<https://devfeed.tech/articles/zerovln-learning-to-navigate-without-supervised-fine-tuning-of-the-vlm-brain-145668.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26414>)

Author: Zixuan Huang, Yang Zhou, Guli Zhang, Sunzhu Li, Hongyan Xie, Zehao Chen, Fuzhen Zhuang, Kaixuan Wang, Xiaozhi Chen, Yikun Ban, Deqing Wang

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Robot Navigation](<https://devfeed.tech/topics/robot-navigation.md>), [vlm](<https://devfeed.tech/topics/vlm.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [capabilities](<https://devfeed.tech/tags/capabilities.md>), [continuous](<https://devfeed.tech/tags/continuous.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [model](<https://devfeed.tech/tags/model.md>), [natural-language](<https://devfeed.tech/tags/natural-language.md>), [navigation](<https://devfeed.tech/tags/navigation.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [sft](<https://devfeed.tech/tags/sft.md>), [tool-calling](<https://devfeed.tech/tags/tool-calling.md>), [vision-language-model](<https://devfeed.tech/tags/vision-language-model.md>)

### AI overview

ZeroVLN targets a tradeoff in vision-language navigation: fine-tuning the VLM brain on navigation data can change its weights and weaken general visual capabilities. Instead, the framework leaves the brain without navigation-domain supervised fine-tuning and gives it a semantic tool-calling interface to send natural-language movement requests to an action expert. The expert is first adapted to execute those requests; closed-loop reinforcement learning then jointly improves the brain's movement decisions and the expert's continuous control. On standard VLN-CE benchmarks, using the same navigation episodes, the authors report substantial gains over representative SFT-based hierarchical methods, with only minor changes to the brain's weights and largely preserved general capabilities.

### Source excerpt

arXiv:2611.26414v1 Announce Type: new Abstract: Representative vision-language navigation (VLN) methods typically couple a vision-language model (VLM) brain with a learnable action head and fine-tune the VLM brain on navigation-domain data to predict geometric goals. However, such adaptation can substantially alter base VLM weights and degrade general visual capabilities. We introduce ZeroVLN, a slow-fast navigation framework that achieves effective navigation with zero navigation-domain SFT of the VLM brain. Rather than teaching the brain to predict geometric goals, we equip it with a semantic tool-calling interface for conveying movement decisions to an action expert in natural language. We first adapt the action expert to carry out these movement requests, then jointly improve the brain's movement decisions and the action expert's continuous execution through closed-loop reinforcement learning (RL). Trained on the same navigation episodes, ZeroVLN substantially outperforms representative SFT-based hierarchical VLN methods on standard VLN-CE (continuous environments) benchmarks, while inducing only minor changes to the brain's weights and largely preserving its general capabilities. To our knowledge, ZeroVLN is the first slow-fast VLN framework with zero navigation-domain SFT of the VLM brain, and the first to jointly optimize its slow and fast policies through closed-loop agentic RL in VLN-CE.

## Persona Lenses: What Makes a Vision-Language Model See Through Its Persona's Eyes?

DevFeed: [Persona Lenses: What Makes a Vision-Language Model See Through Its Persona's Eyes?](<https://devfeed.tech/articles/persona-lenses-what-makes-a-vision-language-model-see-through-its-persona-s-eyes-145526.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25508>)

Author: Yoonho Shin

Published: 2026-11-20T05:00:00Z

Content type: comparison

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [vlm](<https://devfeed.tech/topics/vlm.md>)

Tags: [cost](<https://devfeed.tech/tags/cost.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-hc](<https://devfeed.tech/tags/cs-hc.md>), [language](<https://devfeed.tech/tags/language.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [lenses](<https://devfeed.tech/tags/lenses.md>), [models](<https://devfeed.tech/tags/models.md>), [pages](<https://devfeed.tech/tags/pages.md>), [persona](<https://devfeed.tech/tags/persona.md>), [pixel](<https://devfeed.tech/tags/pixel.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [qwen3](<https://devfeed.tech/tags/qwen3.md>), [qwen3-5](<https://devfeed.tech/tags/qwen3-5.md>), [reward](<https://devfeed.tech/tags/reward.md>), [scale](<https://devfeed.tech/tags/scale.md>), [size](<https://devfeed.tech/tags/size.md>), [training](<https://devfeed.tech/tags/training.md>), [vision](<https://devfeed.tech/tags/vision.md>), [vision-language-model](<https://devfeed.tech/tags/vision-language-model.md>), [vision-language-models](<https://devfeed.tech/tags/vision-language-models.md>), [web](<https://devfeed.tech/tags/web.md>), [weight](<https://devfeed.tech/tags/weight.md>)

### AI overview

The study measures how much persona attributes change a vision-language model's simulated gaze on web pages. It compares three ways to make personas influence gaze: a two-step text lens, interest-matched pixel marks, and a dwell reward constrained to keep scanpaths natural. On Qwen3-VL (4B and 32B) and Qwen3.5-9B, the text lens separates personas strongly but can shift gaze in response to profile changes unrelated to looking behavior. The pixel lens separates them less, while remaining unaffected by unrelated attributes. At 4B and 9B, the weight lens distinguishes personas on interests held out from training and requires no extra step for a new page. The 32B model achieves persona separation with the base prompt alone, so the findings point to a cost and model-size tradeoff. The supplied text is an abstract, not the full paper.

### Source excerpt

arXiv:2611.25508v1 Announce Type: cross Abstract: Vision-language models have been used to simulate where a user looks on a web page. However, a persona in the prompt reaches the simulated gaze only partly. We measure this reach as the difference in one model's gaze between two MatrAIx personas that differ in a single attribute. We introduce persona lenses, which place the persona between the model and the page: a text lens (a two-step prompt in which the model first writes what the persona would look at), a pixel lens (marks on the region matching the persona's interests) and a weight lens (a dwell reward gated to keep the scanpath natural). Using Qwen3-VL (4B and 32B) and Qwen3.5-9B on generated and live pages, we compare the lenses with one another and with scale, the 32B model with no lens. It has been confirmed that the text lens separates the personas strongly but is not robust to profile changes that should not move gaze. The pixel lens separates the personas less but is unaffected by attributes unrelated to gaze. At 4B and 9B, the weight lens separates the personas on interests held out from training, with no extra step on a new page. The 32B model separates the personas with the base prompt alone. Therefore, the choice of a lens depends on cost and model size.

## Goal-Guided Reinforcement Learning via Fine-Tuning Vision-Language Models

DevFeed: [Goal-Guided Reinforcement Learning via Fine-Tuning Vision-Language Models](<https://devfeed.tech/articles/goal-guided-reinforcement-learning-via-fine-tuning-vision-language-models-145097.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25742>)

Author: Yajie Qi, Wei Wei, Lin Li, Lijun Zhang, Jianye Hao

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [vlm](<https://devfeed.tech/topics/vlm.md>), [dpo](<https://devfeed.tech/topics/dpo.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>)

Tags: [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [direct-preference-optimization](<https://devfeed.tech/tags/direct-preference-optimization.md>), [exploration](<https://devfeed.tech/tags/exploration.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [kl-regularization](<https://devfeed.tech/tags/kl-regularization.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [vision-language-models](<https://devfeed.tech/tags/vision-language-models.md>)

### AI overview

The proposed framework fine-tunes a vision-language model on CCGP-2M, a multimodal goal-preference dataset built from Craftax-Classic interaction trajectories, then uses the adapted model to guide online reinforcement learning. Offline fine-tuning uses KLTDPO, combining token-level KL regularization with distributionally robust reweighting. In Craftax-Classic experiments, the method collected diamonds earlier than state-of-the-art baselines, achieved the highest score and reward, and had the highest success rates on most achievements. The source does not provide details about the baselines or quantitative effect sizes.

### Source excerpt

arXiv:2611.25742v1 Announce Type: new Abstract: Exploration remains a major challenge for reinforcement learning (RL) agents in Craftax-style environments due to sparse rewards and long horizons. Large language models (LLMs) can guide exploration by generating high-level goals, but methods using models without explicit task adaptation often rely on carefully designed prompts and may produce inconsistent or contextually inappropriate goals. We introduce CCGP-2M, a multimodal goal-preference dataset constructed from interaction trajectories in Craftax-Classic, and propose a vision-language model (VLM)-guided RL framework that learns environment-specific goal guidance from this dataset. The framework consists of two stages: offline VLM fine-tuning and online goal-guided RL. In the offline stage, we align the VLM with goal preferences using Kullback-Leibler Token-level Direct Preference Optimization (KLTDPO), which combines token-level KL regularization with distributionally robust reweighting. Experiments on Craftax-Classic show that our method collects diamonds earlier than state-of-the-art baselines, achieves the highest score and reward, and attains the highest success rates on most achievements.

## Keel: A Frontier Model Teaches a Vision-Language Model by Rewriting Its Weights

DevFeed: [Keel: A Frontier Model Teaches a Vision-Language Model by Rewriting Its Weights](<https://devfeed.tech/articles/keel-a-frontier-model-teaches-a-vision-language-model-by-rewriting-its-weights-145491.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25257>)

Author: Jiajun Fan, Neil He, Bangji Yang, Qinglong Zheng, Ziwen Wang, Chaoran Cheng, Changyi Yang, Hanqi Mao, Ge Liu

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [vlm](<https://devfeed.tech/topics/vlm.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Frontier Model](<https://devfeed.tech/topics/frontier-model.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [attention](<https://devfeed.tech/tags/attention.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [gui](<https://devfeed.tech/tags/gui.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [qwen3-5](<https://devfeed.tech/tags/qwen3-5.md>), [qwen3-5-0-8b](<https://devfeed.tech/tags/qwen3-5-0-8b.md>), [vision-language-model](<https://devfeed.tech/tags/vision-language-model.md>)

### AI overview

Keel uses a frontier multimodal model as an agent to identify and edit weights causing small vision-language models to loop on image questions. Given the looping symptom, it uses failure reports and attention visualizations to guide edits, then retains edits that reduce failures without lowering accuracy on held-out questions. Independent editors traced loops in Qwen3.5-0.8B and 2B to middle-layer attention outputs, where the smaller model stops consulting the image during a loop. Across six image benchmarks, edits removed most loops, shortened answers, and improved average accuracy over thinking-mode models. The method requires no training and preserves model size and per-token cost; it can also combine with budget forcing when held-out results favor it. The excerpt reports the best average accuracy for Keel on both models with the best available weights, and significant gains over compared training-free baselines on the larger model.

### Source excerpt

arXiv:2611.25257v1 Announce Type: cross Abstract: Can a frontier model repair a vision-language model (VLM) by rewriting its weights? Small reasoning VLMs such as Qwen3.5 need such a repair: in thinking mode they fall into loops on many image questions, repeating themselves until the token limit, spending the whole budget and losing the answer. The usual remedies act on the decoding, leaving the looping model as it was, or must be told what to change. Yet the cause is written in the weights. Keel, the first agentic weight-editing method for VLMs on image tasks, finds and removes it there. Told only the symptom, an editor, a frontier multimodal LLM acting as an agent, locates the cause, edits the weights and tests them, with a failure census and instruments showing it the failing images and where the VLM attends; among several editors, the harness keeps only edits that clearly reduce the failure without lowering accuracy on questions no editor saw. The repair needs no training, keeps the model's size and per-token cost, and applies to already-edited weights. The editors expose where the loops live: from the original weights, independent editors on both models trace them to the attention outputs of the middle layers, and the smaller VLM stops consulting the image inside a loop. On Qwen3.5-0.8B and Qwen3.5-2B, across six image benchmarks from visual mathematics to GUI grounding, the edit removes most loops on every benchmark and shortens answers, and the edited models are significantly more accurate than the thinking models on average; the larger one gains most on GUI grounding, a task absent from the practice questions. Because the repair lives in the weights, it combines with budget forcing where held-out questions favor it: on the best available weights, Keel has the best average accuracy on both models, significantly above every training-free baseline we compare on the larger one.

## VLRS-Bench: A Vision-Language Reasoning Benchmark for Remote Sensing

DevFeed: [VLRS-Bench: A Vision-Language Reasoning Benchmark for Remote Sensing](<https://devfeed.tech/articles/vlrs-bench-a-vision-language-reasoning-benchmark-for-remote-sensing-145314.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2602.07045>)

Author: Zhiming Luo, Hebaixu Wang, Haonan Guo, Jing Zhang, Bo Du

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [vlm](<https://devfeed.tech/topics/vlm.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [remote-sensing](<https://devfeed.tech/tags/remote-sensing.md>)

### AI overview

VLRS-Bench targets a gap in remote-sensing evaluation: existing benchmarks emphasize perception, while the proposed benchmark focuses on complex reasoning. It contains 2,000 question-answer pairs across 14 tasks and the dimensions of Cognition, Decision, and Prediction; questions average 129.87 words and cover up to eight temporal phases. Its construction pipeline combines remote-sensing-specific priors with expert knowledge to support geospatial realism and reasoning complexity. Experiments report significant bottlenecks in existing state-of-the-art multimodal large language models. The supplied text does not identify individual models or detail their results.

### Source excerpt

arXiv:2602.07045v3 Announce Type: replace-cross Abstract: Recent advancements in Multimodal Large Language Models (MLLMs) have enabled complex reasoning. However, existing remote sensing (RS) benchmarks remain heavily biased toward perception tasks, such as object recognition and scene classification. This limitation hinders the development of MLLMs for cognitively demanding RS applications. To address this, we propose a Vision Language ReaSoning Benchmark (VLRS-Bench), which is the first benchmark exclusively dedicated to complex RS reasoning. Structured across the three core dimensions of Cognition, Decision, and Prediction, VLRS-Bench comprises 2,000 question-answer pairs with an average question length of 129.87 words, spanning 14 tasks and up to eight temporal phases. VLRS-Bench is constructed via a specialized pipeline that integrates RS-specific priors and expert knowledge to ensure geospatial realism and reasoning complexity. Experimental results reveal significant bottlenecks in existing state-of-the-art MLLMs, providing critical insights for advancing multimodal reasoning within the remote sensing community.

## Beyond Entity Recognition: Vision-Centric Knowledge Visual Question Answering

DevFeed: [Beyond Entity Recognition: Vision-Centric Knowledge Visual Question Answering](<https://devfeed.tech/articles/beyond-entity-recognition-vision-centric-knowledge-visual-question-answering-145570.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25680>)

Author: Jiatong Ma, Yuchen Liu, Longteng Guo, Yichen Yuan, Yuze Wu, Dongze Hao, Ru Zhen, Yanhao Zhang, Haonan Lu, Jing Liu

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [vlm](<https://devfeed.tech/topics/vlm.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [image recognition](<https://devfeed.tech/topics/image-recognition.md>), [Computer vision](<https://devfeed.tech/topics/computer-vision.md>), [knowledge-engineering](<https://devfeed.tech/topics/knowledge-engineering.md>)

Tags: [acquisition](<https://devfeed.tech/tags/acquisition.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-retrieval](<https://devfeed.tech/tags/agentic-retrieval.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [bottleneck](<https://devfeed.tech/tags/bottleneck.md>), [complex](<https://devfeed.tech/tags/complex.md>), [cross](<https://devfeed.tech/tags/cross.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [external](<https://devfeed.tech/tags/external.md>), [knowledge](<https://devfeed.tech/tags/knowledge.md>), [knowledge-base](<https://devfeed.tech/tags/knowledge-base.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [open-web](<https://devfeed.tech/tags/open-web.md>), [question](<https://devfeed.tech/tags/question.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [scale](<https://devfeed.tech/tags/scale.md>), [vision](<https://devfeed.tech/tags/vision.md>)

### AI overview

VC-VQA benchmarks vision-centric knowledge reasoning in multimodal large language models, including how they acquire and use visual evidence across multiple reasoning stages. It distinguishes internal from external images and pre-retrieval from post-retrieval visual information; tasks include retrieving external visual evidence, spotting clues before retrieval, and localizing, verifying, or matching retrieved knowledge to an input image. Its controlled multimodal knowledge base provides traceable textual and visual evidence. The authors evaluate 21 models across four settings: no external knowledge, provided evidence, knowledge-base retrieval, and open-web search. They report that models still struggle, with fine-grained visual knowledge acquisition the main bottleneck, while agentic retrieval points to the value of multi-step planning and cross-modal integration.

### Source excerpt

arXiv:2611.25680v1 Announce Type: cross Abstract: We propose VC-VQA, a large-scale benchmark for vision-centric knowledge reasoning, aiming to systematically evaluate the capabilities of Multimodal Large Language Models (MLLMs) in fine-grained visual knowledge acquisition and complex multi-stage reasoning. Unlike existing benchmarks that primarily treat visual information as entity recognition and rely on text-based external knowledge, VC-VQA explicitly models the role of visual information by considering both its source (internal vs. external images) and stage of involvement (pre-retrieval vs. post-retrieval). Specifically, VC-VQA requires models to retrieve external visual evidence, identify visual clues before retrieval, and localize, verify, or match retrieved knowledge against the internal image. We further construct a controlled multimodal knowledge base with traceable textual and visual evidence for each sample. We evaluate 21 representative MLLMs under four settings: no external knowledge, with provided evidence, knowledge base retrieval, and open-web search. Experimental results lead to three key findings: current models still struggle with vision-centric knowledge reasoning; fine-grained visual knowledge acquisition remains the primary bottleneck; and agentic retrieval highlights the value of multi-step planning and cross-modal knowledge integration. Overall, VC-VQA provides a more challenging benchmark for evaluating vision-centric knowledge reasoning in MLLMs. Our code and dataset are available at https://github.com/CASIA-IVA-Lab/VC-VQA.

## Efficient Latent Reasoning for Vision-Language Model via Dynamics-Aware Quantization and Distillation

DevFeed: [Efficient Latent Reasoning for Vision-Language Model via Dynamics-Aware Quantization and Distillation](<https://devfeed.tech/articles/efficient-latent-reasoning-for-vision-language-model-via-dynamics-aware-quantization-and-distillation-145567.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25670>)

Author: Yanjing Li, Sheng Xu, Baochang Zhang, Xianbin Cao, Yawei Li

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [vlm](<https://devfeed.tech/topics/vlm.md>), [Chain-of-thought](<https://devfeed.tech/topics/chain-of-thought.md>), [Post-training optimization](<https://devfeed.tech/topics/post-training-optimization.md>)

Tags: [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [distillation](<https://devfeed.tech/tags/distillation.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [vision](<https://devfeed.tech/tags/vision.md>), [vision-language-model](<https://devfeed.tech/tags/vision-language-model.md>)

### AI overview

QL-VLM aims to make latent chain-of-thought reasoning in vision-language models cheaper without losing predictive accuracy under 4-bit quantization. Its Dynamics-Guided Quantization uses nonuniform codebooks to preserve rotary-key structure for information retrieval; Dynamics-Preserving Distillation aligns continuous thoughts, latent-query readouts, and predictions with a full-precision teacher to support later low-precision reasoning. On ScienceQA-IMG with Qwen2-VL-2B-Instruct, the abstract reports 90.43% accuracy, a 40.33% improvement over baseline, 16.64x fewer bit-operations, and 6.40x lower measured latency than explicit CoT.

### Source excerpt

arXiv:2611.25670v1 Announce Type: new Abstract: Efficient chain-of-thought (CoT) reasoning in vision-language models (VLMs) requires reducing both intermediate reasoning steps and their execution cost. Latent CoT methods compress textual rationales into continuous thoughts, yet their full-precision (FP) models retain substantial storage and computation costs. Lowering numerical precision could reduce these remaining costs, yet direct quantization severely degrades reasoning accuracy, particularly with 4-bit weights and activations. Operator and perturbation analyses reveal key sensitivity and direction-dependent predictive effects of continuous-thought errors, motivating preservation of information retrieval and subsequent reasoning beyond reconstruction accuracy. We therefore introduce QL-VLM, a dynamics-aware Quantization framework for efficient Latent reasoning in Vision-Language Models (VLMs). First, Dynamics-Guided Quantization (DGQ) uses nonuniform codebooks that preserve rotary-key structure, mitigating quantization-induced distortion in vision-language information retrieval. Second, Dynamics-Preserving Distillation (DPD) aligns continuous thoughts, latent-query readouts, and predictions with a full-precision teacher, maintaining predictive consistency through subsequent low-precision reasoning. On ScienceQA-IMG with Qwen2-VL-2B-Instruct, 4-bit QL-VLM achieves 90.43% accuracy and improves baseline by 40.33%, with 16.64x fewer bit-operations and 6.40x lower measured latency than explicit CoT.

## ChronoPatch: Black-Box Energy-Latency Attacks on Vision-Language Models

DevFeed: [ChronoPatch: Black-Box Energy-Latency Attacks on Vision-Language Models](<https://devfeed.tech/articles/chronopatch-black-box-energy-latency-attacks-on-vision-language-models-145634.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26117>)

Author: Jiawei Xing, Li Wang, Xiangtao Meng, Chuanchao Zang, Rui Wen, Zheng Li

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [vlm](<https://devfeed.tech/topics/vlm.md>), [Adversarial attacks](<https://devfeed.tech/topics/adversarial-attacks.md>), [LLM security](<https://devfeed.tech/topics/llm-security.md>)

Tags: [adversarial](<https://devfeed.tech/tags/adversarial.md>), [attacks](<https://devfeed.tech/tags/attacks.md>), [availability](<https://devfeed.tech/tags/availability.md>), [black-box](<https://devfeed.tech/tags/black-box.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [energy](<https://devfeed.tech/tags/energy.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [latency](<https://devfeed.tech/tags/latency.md>), [vision-language-models](<https://devfeed.tech/tags/vision-language-models.md>)

### AI overview

ChronoPatch shows that visual inputs can prolong vision-language model generation even when an attacker sees only the final response. It uses response-length-guided evolutionary search to find visual delay anchors, then subspace-calibrated decision-based optimization to reduce perturbation magnitude. Under a 5% spatial budget, tests on four open-source VLMs report latency increases of 47.86%-293.34% and GPU energy increases of 41.44%-249.70%, alongside more repetitive outputs. On three commercial VLM APIs, with 500 queries per API, response length rose 37.6%-81.5% and client-observed latency reached 1.82-2.29 times baseline. The abstract identifies this as a generation-based availability risk and motivates safeguards that account for generation.

### Source excerpt

arXiv:2611.26117v1 Announce Type: cross Abstract: Vision-language models (VLMs) are increasingly deployed as generative inference services, where autoregressive decoding makes response length a key determinant of latency and computational cost. This creates an availability risk: adversarial visual inputs can prolong generation and increase serving overhead. Existing energy-latency attacks typically rely on gradients, logits, or token-level feedback, leaving open whether such risks persist when an attacker observes only the final response. We present ChronoPatch, a black-box attack that induces prolonged VLM generation requiring only the final response. It discovers visual delay anchors through response-length-guided evolutionary search, then minimizes perturbation magnitude through subspace-calibrated decision-based optimization. Across four open-source VLMs, ChronoPatch increases inference latency by 47.86%--293.34% and GPU energy consumption by 41.44%--249.70% under a 5% spatial budget. It also induces more repetitive outputs, as reflected by increased n-gram repetition ratios. On three commercial VLM APIs, ChronoPatch increases response length by 37.6%--81.5% and client-observed latency by 1.82x--2.29x under query budgets of 500 per API. These results reveal a generation-based availability risk that persists under final-response-only access and motivate generation-aware safeguards for deployed generative AI services.

## Let a Small Model Read First: Faster Prefill for Long Interleaved Image-Text Inputs

DevFeed: [Let a Small Model Read First: Faster Prefill for Long Interleaved Image-Text Inputs](<https://devfeed.tech/articles/let-a-small-model-read-first-faster-prefill-for-long-interleaved-image-text-inputs-142796.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24431>)

Author: Xiaolin Lin, Kai-Chiang Wu, Bing Li, Grace Li Zhang

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [vlm](<https://devfeed.tech/topics/vlm.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>)

Tags: [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [image](<https://devfeed.tech/tags/image.md>), [prefill](<https://devfeed.tech/tags/prefill.md>), [tokens](<https://devfeed.tech/tags/tokens.md>), [vision-language-models-vlms](<https://devfeed.tech/tags/vision-language-models-vlms.md>)

### AI overview

SkimPrefill reduces long-input prefill time by having a small hybrid model scan the full image-text input up to an early layer, using half-resolution images, then rank whole units such as images or paragraphs with one attention head. The target model receives the highest-ranked units as a shorter prompt. The abstract reports that one untrained selector works across model families. On MMLongBench at 128K tokens, retaining 10% of input tokens sped up prefill by 3.7x for Qwen3.5-9B, 11.6x for Gemma-4-12B, and 8.2x for Qwen3-VL-32B while maintaining average accuracy.

### Source excerpt

arXiv:2611.24431v1 Announce Type: cross Abstract: Vision-language models (VLMs) can answer questions about long interleaved inputs, such as documents of many pages and collections of hundreds of images, but they are slow to start: on inputs of about 100K tokens, the prefill takes most of the wait for the first output token. A common approach prunes the input by scoring its tokens with the target model itself or with a small model of the same family that first writes an answer. However, this scoring takes much of the time that pruning saves. We find that the evidence for a question lies in a few whole units (images or paragraphs), and that a small hybrid model finds them with one attention head of an early layer, even at half image resolution. Based on these findings, we propose SkimPrefill, which lets the small hybrid model read the whole input first, up to that early layer, with images at half resolution. Most layers of the small hybrid model (Qwen3.5-0.8B) run in linear time, so this pass is short. SkimPrefill then ranks every unit with that head and gives the target the top units as an ordinary shorter prompt. One untrained selector serves targets of different model families. Extensive experiments on MMLongBench at 128K tokens, with 19 datasets of five task types, show the effectiveness of SkimPrefill. With 10% of the input tokens kept, SkimPrefill speeds up the prefill of Qwen3.5-9B, Gemma-4-12B and Qwen3-VL-32B by 3.7 times, 11.6 time and 8.2 times while maintaining their average accuracy.

## SixthView: A Benchmark for Hidden-View Reconstruction and 3D Assembly Reasoning

DevFeed: [SixthView: A Benchmark for Hidden-View Reconstruction and 3D Assembly Reasoning](<https://devfeed.tech/articles/sixthview-a-benchmark-for-hidden-view-reconstruction-and-3d-assembly-reasoning-142719.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23891>)

Author: Pooya Khosravi, Jen-Feng Chang, Fnu Pramono

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [3D](<https://devfeed.tech/topics/3d.md>), [vlm](<https://devfeed.tech/topics/vlm.md>)

Tags: [3d](<https://devfeed.tech/tags/3d.md>), [assembly](<https://devfeed.tech/tags/assembly.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [computer-science](<https://devfeed.tech/tags/computer-science.md>), [computer-vision](<https://devfeed.tech/tags/computer-vision.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [github](<https://devfeed.tech/tags/github.md>), [grid](<https://devfeed.tech/tags/grid.md>), [hidden](<https://devfeed.tech/tags/hidden.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [recovery](<https://devfeed.tech/tags/recovery.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>), [view](<https://devfeed.tech/tags/view.md>), [vision-language-models](<https://devfeed.tech/tags/vision-language-models.md>)

### AI overview

SixthView tests whether models can reconstruct an unambiguous hidden orthographic view of a 3D polycube assembly from five views and component-shape information. It includes 602 synthetic assemblies, with the target represented as a dense grid of part labels; the authors verify that each item's exact geometry and observed labels determine a unique target. On 552 fully evaluated items, exact-grid accuracy with images plus exact text ranged from 33.2% to 85.4% across three vision-language models. Adding exact text to separate images improved accuracy by 19.7-63.0 percentage points, while adding images to exact text changed it by only −0.5 to +2.1 points. Separate image panels scored 6.7-11.1 points above a merged sheet. The abstract attributes the image-only variants' added challenge to recovering shapes from occluded renderings.

### Source excerpt

arXiv:2611.23891v1 Announce Type: cross Abstract: Inferring the unseen side of a 3D assembly requires reasoning about part placement and occlusion. Different assemblies can match the observed views while producing different hidden views, leaving the target ambiguous. We introduce SixthView, a benchmark in which models predict a sixth orthographic view from five observed views and information about the component shapes. The benchmark contains 602 synthetic polycube assemblies with 2--10 parts and requires a dense grid of part labels for the missing view, allowing both exact-view and cell-level evaluation. For each item, we exhaustively verify that all legal assemblies consistent with the exact component geometry and observed label grids yield the same target view. With these inputs, the task tests symbolic assembly and coordinate reasoning. The image-only variants additionally require shape recovery from rendered views, where occlusion can make different 3D shapes look identical. We evaluate three vision-language models under four input conditions. Across 552 items with complete evaluations, exact-grid accuracy with images plus exact text ranges from 33.2% to 85.4%. Adding exact text to separate images improves accuracy by 19.7--63.0 percentage points, whereas adding images to exact text changes it by only $-0.5$ to $+2.1$ points. Separate image panels outperform a merged sheet by 6.7--11.1 points. The results show that performance depends strongly on access to exact symbolic information, with little further improvement from images once the geometry is specified. GitHub: https://github.com/facebookresearch/sixthview_benchmark

## GRAFT: Graph-Regularized Prototype Fields for Realistic Test-Time Adaptation of VLMs

DevFeed: [GRAFT: Graph-Regularized Prototype Fields for Realistic Test-Time Adaptation of VLMs](<https://devfeed.tech/articles/graft-graph-regularized-prototype-fields-for-realistic-test-time-adaptation-of-vlms-143408.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23591>)

Author: Maxx Richard Rahman, Wolfgang Maass

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [vlm](<https://devfeed.tech/topics/vlm.md>), [few-shot](<https://devfeed.tech/topics/few-shot.md>)

Tags: [anchor](<https://devfeed.tech/tags/anchor.md>), [bayes](<https://devfeed.tech/tags/bayes.md>), [class](<https://devfeed.tech/tags/class.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [features](<https://devfeed.tech/tags/features.md>), [fields](<https://devfeed.tech/tags/fields.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [graph](<https://devfeed.tech/tags/graph.md>), [inference](<https://devfeed.tech/tags/inference.md>), [markov](<https://devfeed.tech/tags/markov.md>), [model](<https://devfeed.tech/tags/model.md>), [online](<https://devfeed.tech/tags/online.md>), [protocols](<https://devfeed.tech/tags/protocols.md>), [prototype](<https://devfeed.tech/tags/prototype.md>), [prototypes](<https://devfeed.tech/tags/prototypes.md>), [semantic](<https://devfeed.tech/tags/semantic.md>), [test-time-adaptation](<https://devfeed.tech/tags/test-time-adaptation.md>), [vision-language-model](<https://devfeed.tech/tags/vision-language-model.md>), [vlms](<https://devfeed.tech/tags/vlms.md>), [zero-shot](<https://devfeed.tech/tags/zero-shot.md>)

### AI overview

GRAFT addresses cases where test-time adaptation of vision-language models underperforms zero-shot classification because batches contain few classes or online streams are correlated. It represents adapted prototypes as text anchors plus class-specific displacements, partially pooled across a semantic class graph using a Gaussian Markov random field prior, with an explicit model of class presence. Inference is closed-form and gradient-free on cached features. The prior spans anchor-only shrinkage and a single global shift; absent classes are harmonic extensions of present classes, and pooling strength can be selected by empirical Bayes. The abstract reports that GRAFT improves on zero-shot classification across realistic batch and online protocols on eleven benchmarks and four ImageNet shifts, and beats anchor-based transduction when few classes are present.

### Source excerpt

arXiv:2611.23591v1 Announce Type: cross Abstract: Unlabeled test data can improve the zero-shot classifier of a vision-language model, yet transductive and test-time adaptation methods often fall below the zero-shot baseline when only a few vocabulary classes are present in a batch or when streams are correlated. We trace this failure to a shared design: every class owns independent parameters estimated from its own samples, so absent or rare classes are left at biased text anchors or fitted to foreign samples. We propose Graph-Regularized Anchor-displacement Field Transduction (GRAFT), which models the adapted prototypes as text anchors plus a displacement field that is partially pooled over a semantic class graph through a Gaussian Markov random field prior, together with an explicit class-presence model. Inference is closed-form and gradient-free on cached features. Anchor-only shrinkage and a single global shift are the two extreme settings of the prior; absent classes are provably harmonic extensions of the present ones, and the pooling strength can be chosen by empirical Bayes. Under the realistic batch and online protocols on eleven benchmarks and four ImageNet shifts, GRAFT improves over the zero-shot model in every setting and over anchor-based transduction where few classes are present.

## SEE: Unlocking the Hidden Perception of Frozen VLMs

DevFeed: [SEE: Unlocking the Hidden Perception of Frozen VLMs](<https://devfeed.tech/articles/see-unlocking-the-hidden-perception-of-frozen-vlms-142731.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23962>)

Author: Alin-Ionut Popa, Tuo Zhang, Dimitrios Dimitriadis

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [vlm](<https://devfeed.tech/topics/vlm.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [computer-vision](<https://devfeed.tech/tags/computer-vision.md>), [confidence](<https://devfeed.tech/tags/confidence.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [image](<https://devfeed.tech/tags/image.md>), [infographic](<https://devfeed.tech/tags/infographic.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [ocr](<https://devfeed.tech/tags/ocr.md>), [small](<https://devfeed.tech/tags/small.md>), [text](<https://devfeed.tech/tags/text.md>), [training](<https://devfeed.tech/tags/training.md>), [transcript](<https://devfeed.tech/tags/transcript.md>), [vision](<https://devfeed.tech/tags/vision.md>), [vision-language-models](<https://devfeed.tech/tags/vision-language-models.md>), [vision-language-models-vlms](<https://devfeed.tech/tags/vision-language-models-vlms.md>), [vlm](<https://devfeed.tech/tags/vlm.md>), [vlms](<https://devfeed.tech/tags/vlms.md>), [voting](<https://devfeed.tech/tags/voting.md>)

### AI overview

Self-Elicited Evidence (SEE) helps frozen vision-language models recover fine image details without training, detectors, or OCR. When an initial answer has low confidence, the model proposes regions to inspect, answers from enlarged crops and its own transcript, then combines those answers with its original prediction using confidence-weighted voting. The abstract reports 2.3-13.1 point gains across every small-object benchmark tested on five backbones, and says SEE outperforms five training-free baselines on small objects. With SEE, Qwen3.5-4B reportedly exceeds the direct-answer performance of Qwen3.5-9B and matches Qwen3.5-27B, which is six times larger. These claims reflect the abstract's reported evaluations.

### Source excerpt

arXiv:2611.23962v1 Announce Type: cross Abstract: Vision-language models (VLMs) grasp the overall content of an image but often miss the fine details a question hinges on, such as a small object in a large photo or a number in a dense infographic. Yet even when their answer is wrong, the region they point to often contains the evidence. We introduce Self-Elicited Evidence (SEE), a training-free method that lets a frozen VLM look again where it points. When the initial answer has low confidence, SEE prompts the model to propose regions where the details may appear, answers the question on these enlarged crops, and aggregates the answers with the original prediction through confidence-weighted voting. The model also answers with its own transcript, which helps on text-rich images. SEE needs no detector or OCR engine. Across five backbones, SEE improves every small-object benchmark by 2.3-13.1 points and outperforms five training-free baselines on small objects. With SEE, Qwen3.5-4B surpasses the direct answer of Qwen3.5-9B and matches the 6 times larger Qwen3.5-27B.

## Assessing Vision-Language Models for Early Prediction of Unsafe Situations in Robotics

DevFeed: [Assessing Vision-Language Models for Early Prediction of Unsafe Situations in Robotics](<https://devfeed.tech/articles/assessing-vision-language-models-for-early-prediction-of-unsafe-situations-in-robotics-142205.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24146>)

Author: Muhammad Farooq, Aitor Arrieta, Paolo Arcaini, Shaukat Ali, Thomas Peyrucain

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [vlm](<https://devfeed.tech/topics/vlm.md>), [qwen](<https://devfeed.tech/topics/qwen.md>), [Uncertainty quantification LLMs](<https://devfeed.tech/topics/uncertainty-quantification-llms.md>), [Monitoring & Alerting](<https://devfeed.tech/topics/monitoring-alerting.md>)

Tags: [classification](<https://devfeed.tech/tags/classification.md>), [cs-ro](<https://devfeed.tech/tags/cs-ro.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [early-warning](<https://devfeed.tech/tags/early-warning.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [history](<https://devfeed.tech/tags/history.md>), [llava](<https://devfeed.tech/tags/llava.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [performance](<https://devfeed.tech/tags/performance.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [qwen](<https://devfeed.tech/tags/qwen.md>), [robot-safety](<https://devfeed.tech/tags/robot-safety.md>), [robotics](<https://devfeed.tech/tags/robotics.md>), [support](<https://devfeed.tech/tags/support.md>), [uncertainty](<https://devfeed.tech/tags/uncertainty.md>), [video](<https://devfeed.tech/tags/video.md>), [vision-language-models](<https://devfeed.tech/tags/vision-language-models.md>), [vision-language-models-vlms](<https://devfeed.tech/tags/vision-language-models-vlms.md>)

### AI overview

The abstract reports an empirical evaluation of Qwen, InternVL, and LLaVA-OneVision for early robot-safety monitoring across HuRoN, Pal Robotics, and CrowdBot, using video and odometry to classify situations as safe, potentially unsafe, or unsafe and assess uncertainty. Qwen had the strongest overall classification and prediction-reliability performance; InternVL showed stronger early-warning capability in several configurations. The effect of robot-state information and temporal history varied by model and evaluation objective, so the preferred configuration depends on whether accurate safety-state classification, reliable uncertainty estimates, or timely warning is the priority.

### Source excerpt

arXiv:2611.24146v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) process visual information and textual instructions and, by using their ability to understand and reason about context, are well-suited for predicting robot failures before they occur. However, evidence supporting this remains limited. To address this, we present an empirical evaluation of VLMs for predicting robot failures. In particular, we experiment with Qwen, InternVL, and LLaVA-OneVision across three robotic case studies (HuRoN, Pal Robotics, and CrowdBot), analyzing both video and odometry data for early failure warning (i.e., before an unsafe situation occurs). Specifically, we evaluate their performance in classifying situations as safe, potentially unsafe, and unsafe, together with uncertainty quantified using various metrics, and study the relationship between uncertainty and performance. Our results showed that Qwen provides the strongest overall classification and prediction-reliability performance, while InternVL shows stronger early-warning capability in several configurations. The results further show that the effect of robot-state information and temporal history depends on the model and evaluation objective. Overall, our findings indicate that general-purpose VLMs can support early robot-safety monitoring, but the most suitable configuration depends on whether the priority is accurate assessment of safety-state, reliable confidence and uncertainty, or timely warning before unsafe events.

## ProtoBridge-CLIP: Observation-Masked Finding Prototypes for Medical Image--Report Representation Learning

DevFeed: [ProtoBridge-CLIP: Observation-Masked Finding Prototypes for Medical Image--Report Representation Learning](<https://devfeed.tech/articles/protobridge-clip-observation-masked-finding-prototypes-for-medical-image-report-representation-learning-142738.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24002>)

Author: Jinrui Wu, Yun Peng

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Medical imaging](<https://devfeed.tech/topics/medical-imaging.md>), [Representation learning](<https://devfeed.tech/topics/representation-learning.md>), [vlm](<https://devfeed.tech/topics/vlm.md>), [image recognition](<https://devfeed.tech/topics/image-recognition.md>)

Tags: [alignment](<https://devfeed.tech/tags/alignment.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [clip](<https://devfeed.tech/tags/clip.md>), [component](<https://devfeed.tech/tags/component.md>), [concepts](<https://devfeed.tech/tags/concepts.md>), [contrastive-learning](<https://devfeed.tech/tags/contrastive-learning.md>), [controls](<https://devfeed.tech/tags/controls.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [framework](<https://devfeed.tech/tags/framework.md>), [global](<https://devfeed.tech/tags/global.md>), [image](<https://devfeed.tech/tags/image.md>), [inspect](<https://devfeed.tech/tags/inspect.md>), [learning](<https://devfeed.tech/tags/learning.md>), [local](<https://devfeed.tech/tags/local.md>), [localization](<https://devfeed.tech/tags/localization.md>), [medical](<https://devfeed.tech/tags/medical.md>), [objectives](<https://devfeed.tech/tags/objectives.md>), [prototypes](<https://devfeed.tech/tags/prototypes.md>), [report](<https://devfeed.tech/tags/report.md>), [representation-learning](<https://devfeed.tech/tags/representation-learning.md>)

### AI overview

ProtoBridge-CLIP separates affirmed, negated, uncertain, and unmentioned report concepts during medical image-report learning. It combines symmetric global image-report alignment with finding supervision that aligns text-derived finding prototypes to local visual tokens, excluding uncertain and unmentioned concepts from direct finding labels, plus a soft relation objective over affirmed concepts. Across three CT and five chest-radiograph benchmarks, it reports macro-AUROCs ranging from 66.5% to 91.4%; the findings and relation objectives consistently improve on matched global-only controls. The abstract also reports ablations and retrieval, localization, robustness, calibration, and per-finding analyses, but does not provide their detailed results.

### Source excerpt

arXiv:2611.24002v1 Announce Type: cross Abstract: Radiology reports distinguish affirmed findings, explicit negations, and unmentioned concepts, but conventional global image--report contrastive learning often compresses these different evidence states into a single study-level target. We introduce \model{}, which combines symmetric global image--report alignment with observation-masked finding supervision and a soft relation objective over affirmed report concepts. Text-derived finding prototypes are aligned with local visual tokens, while uncertain and unmentioned concepts are excluded from direct finding labels. We evaluate the framework on three CT benchmarks and five chest-radiograph benchmarks, together with component ablations, paired and clinical retrieval, localization, perturbation robustness, calibration, and per-finding analyses. \model{} achieves macro-AUROC values of 73.6%, 66.5\%, and 68.8% on CT-RATE, RAD-ChestCT, and INSPECT, and 80.2%, 91.4%, 78.5%, 86.5%, and 90.8% on ChestXray14, CheXpert, PadChest, RSNA, and SIIM-ACR, respectively. Relative to the matched global-only controls, the findings and relation objectives yield consistent gains across the evaluated datasets. These results demonstrate the practical value of treating observed and unobserved report concepts differently during medical image--text representation learning.

## DetectViT: Test-time Backdoor Detection for Vision Transformers via Inter-Head Attention Discrepancy

DevFeed: [DetectViT: Test-time Backdoor Detection for Vision Transformers via Inter-Head Attention Discrepancy](<https://devfeed.tech/articles/detectvit-test-time-backdoor-detection-for-vision-transformers-via-inter-head-attention-discrepancy-143135.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23770>)

Author: Siquan Huang, Yijiang Li, Xingfu Yan, Ningzhi Gao, Leyu Shi, Ying Gao

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [Transformer](<https://devfeed.tech/topics/transformer.md>), [vlm](<https://devfeed.tech/topics/vlm.md>), [image recognition](<https://devfeed.tech/topics/image-recognition.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [backdoor](<https://devfeed.tech/tags/backdoor.md>), [classification](<https://devfeed.tech/tags/classification.md>), [clip](<https://devfeed.tech/tags/clip.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [deit](<https://devfeed.tech/tags/deit.md>), [detection](<https://devfeed.tech/tags/detection.md>), [embeddings](<https://devfeed.tech/tags/embeddings.md>), [entropy](<https://devfeed.tech/tags/entropy.md>), [false-positive](<https://devfeed.tech/tags/false-positive.md>), [false-positive-rate](<https://devfeed.tech/tags/false-positive-rate.md>), [hijacking](<https://devfeed.tech/tags/hijacking.md>), [image](<https://devfeed.tech/tags/image.md>), [image-classification](<https://devfeed.tech/tags/image-classification.md>), [inference](<https://devfeed.tech/tags/inference.md>), [llava](<https://devfeed.tech/tags/llava.md>), [model](<https://devfeed.tech/tags/model.md>), [test](<https://devfeed.tech/tags/test.md>), [transformers](<https://devfeed.tech/tags/transformers.md>), [vision](<https://devfeed.tech/tags/vision.md>), [vision-transformers](<https://devfeed.tech/tags/vision-transformers.md>)

### AI overview

DetectViT detects test-time backdoors in Vision Transformers by measuring two attention anomalies: unusually high spatial overlap between heads and unusually variable attention concentration across heads. It combines these as inter-head consistency and entropy-variance scores, setting thresholds with a small OOD calibration set. The method needs no training data, model outputs, or prior trigger knowledge, and reuses attention weights already computed at inference. The abstract reports results across four attacks, DeiT, CLIP, and LLaVA, and classification and captioning tasks; with 64-128 calibration samples, it reports 100% true positive and 0.17% false positive rates on BadCLIP. These are abstract-reported results, not evidence of performance beyond the stated settings.

### Source excerpt

arXiv:2611.23770v1 Announce Type: cross Abstract: Vision Transformers (ViTs) now underpin the visual encoders of most multimodal models, but this success comes at a cost: relying on third-party pretrained checkpoints leaves these systems open to backdoor attacks. Existing test-time detection methods rely on either the predicted label or intermediate embeddings, limiting their applicability across diverse tasks and incurring substantial computational overhead. In this work, we study how backdoor triggers affect the multi-head attention mechanism of ViTs and identify two distinct anomalous patterns: $\textit{complete attention hijacking}$, causing the attention to exhibit abnormally high inter-head spatial overlap, and $\textit{partial attention hijacking}$, causing an abnormally large variance in the concentration of the attention distribution. Motivated by these findings, we propose $\textbf{DetectViT}$, a test-time backdoor detection method that requires neither training data, model outputs, nor any prior knowledge of the trigger. DetectViT quantifies the two hijacking phenomena via an inter-head consistency score and an inter-head entropy variance score, with thresholds estimated from a small out-of-distribution (OOD) calibration set drawn from an arbitrary source. Across four representative attacks, diverse architectures (DeiT, CLIP, and LLaVA), and downstream tasks (image classification and captioning), DetectViT substantially outperforms all baselines with as few as 64--128 OOD calibration samples, achieving a true positive rate of $100\%$ and a false positive rate of $0.17\%$ on BadCLIP. Notably, it introduces negligible additional overhead by relying solely on attention weights already computed during inference. Code is available at: https://github.com/siquanhuang/DetectViT.

## MoMoVL: Modality-Aware Adaptive Compression for On-Device Sparsely-Activated Vision-Language Models

DevFeed: [MoMoVL: Modality-Aware Adaptive Compression for On-Device Sparsely-Activated Vision-Language Models](<https://devfeed.tech/articles/momovl-modality-aware-adaptive-compression-for-on-device-sparsely-activated-vision-language-models-143435.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23695>)

Author: Dongkeun Lee, Wonjun Lee

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [vlm](<https://devfeed.tech/topics/vlm.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>)

Tags: [activation](<https://devfeed.tech/tags/activation.md>), [adaptive](<https://devfeed.tech/tags/adaptive.md>), [compression](<https://devfeed.tech/tags/compression.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [device](<https://devfeed.tech/tags/device.md>), [edge](<https://devfeed.tech/tags/edge.md>), [edge-devices](<https://devfeed.tech/tags/edge-devices.md>), [inference](<https://devfeed.tech/tags/inference.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [memory](<https://devfeed.tech/tags/memory.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [moe](<https://devfeed.tech/tags/moe.md>), [on-device](<https://devfeed.tech/tags/on-device.md>), [precision](<https://devfeed.tech/tags/precision.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [sparsity](<https://devfeed.tech/tags/sparsity.md>), [tokens](<https://devfeed.tech/tags/tokens.md>), [trade](<https://devfeed.tech/tags/trade.md>), [vision](<https://devfeed.tech/tags/vision.md>), [vision-language-models](<https://devfeed.tech/tags/vision-language-models.md>)

### AI overview

MoMoVL assigns mixed-precision bit-widths in MoE vision-language models using both per-expert activation profiles and each modality's quantization sensitivity. The approach accounts for differences between visual and text tokens, which can affect expert quantization. The abstract reports a better accuracy-compression trade-off than state-of-the-art mixed-precision baselines and up to 15.44% lower memory use than a uniform-precision baseline, with a modest accuracy drop. The abstract does not specify evaluation details or the size of that drop.

### Source excerpt

arXiv:2611.23695v1 Announce Type: cross Abstract: The rise of the agentic web has been accompanied by growing demand for on-device Vision-Language Models (VLMs) that perform diverse cross-modal web tasks with enhanced privacy and reduced latency. Mixture-of-Experts (MoE) models are widely adopted in modern VLMs for compute-efficient scalability, yet their multi-expert architecture poses deployment challenges due to substantial memory demands. Mixed-precision quantization is an effective compression technique that reduces bit-widths while adapting to the heterogeneous hardware budgets of edge devices, particularly well-suited for MoE due to its inherent structural sparsity, which causes imbalanced importance among experts. While prior work has applied quantization for efficient MoE inference at the edge, we point out that in MoE-based VLMs, the modality-specific characteristics of visual and text tokens further impact expert quantization. To address this issue, we present MoMoVL, a modality-aware adaptive compression framework for MoE-based VLMs that brings vision-language inference within reach of on-device resource budgets. Motivated by our empirical findings on modality-induced disparities across experts, MoMoVL jointly exploits the per-expert activation profile and per-modality quantization sensitivity to guide bit-width assignment. Experimental results show that MoMoVL improves the accuracy-compression trade-off relative to state-of-the-art mixed-precision baselines and reduces memory footprint by up to 15.44% with a modest drop in accuracy over its uniform-precision baseline. Our code and data are publicly available at https://github.com/sco-edge/MoMoVL.

[Next page](<https://devfeed.tech/topics/vlm.md?cursor=WyIyMDI2LTExLTE5VDA1OjAwOjAwKzAwOjAwIiwgIjdhNzllZDg2LTRkNzMtNDY0OS04NDEzLWU0OTU1NDJiNDRkMCJd>)