# ModalSearcher: Weaving Visual Evidence into Autonomous Cross-Modal Deep Search

DevFeed: [ModalSearcher: Weaving Visual Evidence into Autonomous Cross-Modal Deep Search](<https://devfeed.tech/articles/modalsearcher-weaving-visual-evidence-into-autonomous-cross-modal-deep-search-118427.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.06131>)

Author: Zhihan Yin, Ruyi Li, Chengyu Fang, Yifeng Yao, Nonghai Zhang, Pengfei Xiong, Terry Zhang, Huishuai Zhang, Dongyan Zhao

Published: 2026-11-05T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>), [agentic workflows](<https://devfeed.tech/topics/agentic-workflows.md>), [Agentic AI](<https://devfeed.tech/topics/what-is-agentic-ai.md>), [AI-driven information retrieval](<https://devfeed.tech/topics/ai-driven-information-retrieval.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [autonomous](<https://devfeed.tech/tags/autonomous.md>), [bench](<https://devfeed.tech/tags/bench.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [complex](<https://devfeed.tech/tags/complex.md>), [construction](<https://devfeed.tech/tags/construction.md>), [cross](<https://devfeed.tech/tags/cross.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [deep](<https://devfeed.tech/tags/deep.md>), [environment](<https://devfeed.tech/tags/environment.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [graph](<https://devfeed.tech/tags/graph.md>), [image](<https://devfeed.tech/tags/image.md>), [image-search](<https://devfeed.tech/tags/image-search.md>), [knowledge](<https://devfeed.tech/tags/knowledge.md>), [knowledge-graph](<https://devfeed.tech/tags/knowledge-graph.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [modal](<https://devfeed.tech/tags/modal.md>), [search](<https://devfeed.tech/tags/search.md>)

## AI overview

ModalSearcher lets an agent choose and combine direct reasoning, text search, text-to-image search, and reverse-image search as its information needs change across multiple hops. The ModalSearch-Pipeline builds interleaved multimodal questions from knowledge-graph paths, obscuring entity names and image targets that could reveal shortcuts while retaining disambiguating clues. The agent is trained with supervised fine-tuning and reinforcement learning in a live Web environment. On five multimodal knowledge-intensive QA benchmarks, ModalSearcher-8B improves average accuracy by 16.2 points over Qwen3-VL-8B under the same agentic workflow; the abstract also reports consistent gains on independently constructed active visual-search benchmarks.

## Source excerpt

arXiv:2611.06131v1 Announce Type: new Abstract: Static and incomplete parametric knowledge limits the performance of multimodal large language models on novel-knowledge and knowledge-intensive tasks, especially those requiring multi-hop evidence acquisition. Existing multimodal agentic search methods often fix visual evidence at the input or a predefined trajectory position, making it difficult for agents to select search modalities as their information needs evolve and to handle complex real-world queries. We define this capability as autonomous cross-modal search: an agent chooses among direct reasoning, text-to-text search, text-to-image search, and reverse-image search, actively introduces visual evidence when appropriate, and switches between and integrates modalities across multiple hops. We propose ModalSearch-Pipeline, a scalable procedure for constructing an interleaved multimodal knowledge graph. It samples paths with diverse depths and modality compositions and uses backward question construction to obscure entity names and image targets that would expose reasoning shortcuts while retaining necessary disambiguating clues, yielding text-only, image-initialized, and arbitrarily interleaved multi-hop problems. We train the ModalSearcher Agent with supervised fine-tuning and reinforcement learning in a live Web environment, and construct ModalSearch-Bench. Experiments show that ModalSearcher-8B improves average accuracy by 16.2 points over Qwen3-VL-8B under the same agentic workflow across five multimodal knowledge-intensive QA benchmarks, with consistent gains on independently constructed active visual-search benchmarks.