# multimodal-ai

A machine-learning approach that processes and integrates multiple data modalities, such as text, images, audio, and video.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Depth Pro 2: Sharp and Temporally Consistent Metric Monocular Depth

DevFeed: [Depth Pro 2: Sharp and Temporally Consistent Metric Monocular Depth](<https://devfeed.tech/articles/depth-pro-2-sharp-and-temporally-consistent-metric-monocular-depth-145199.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26603>)

Author: Peiyun Hu, Marko Mihajlovic, Lyndon Duong, Lahav Lipson, Wei Dong, Ama\\"el Delaunoy, Stephan R. Richter, Vladlen Koltun

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Image](<https://devfeed.tech/topics/image.md>), [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>), [3D](<https://devfeed.tech/topics/3d.md>)

Tags: [code](<https://devfeed.tech/tags/code.md>), [consistency](<https://devfeed.tech/tags/consistency.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [depth-estimation](<https://devfeed.tech/tags/depth-estimation.md>), [frontier](<https://devfeed.tech/tags/frontier.md>), [images](<https://devfeed.tech/tags/images.md>), [maps](<https://devfeed.tech/tags/maps.md>), [metric](<https://devfeed.tech/tags/metric.md>), [models](<https://devfeed.tech/tags/models.md>), [native](<https://devfeed.tech/tags/native.md>), [network](<https://devfeed.tech/tags/network.md>), [runtime](<https://devfeed.tech/tags/runtime.md>), [sharp](<https://devfeed.tech/tags/sharp.md>), [temporal](<https://devfeed.tech/tags/temporal.md>), [trade](<https://devfeed.tech/tags/trade.md>), [train](<https://devfeed.tech/tags/train.md>), [video](<https://devfeed.tech/tags/video.md>)

### AI overview

Depth Pro 2 is a scalable network for estimating dense metric depth from images and video at native resolution. The authors identify a common tradeoff in prior monocular methods: multi-frame approaches tend to favor temporal consistency, while single-frame approaches favor sharp boundaries. Their model processes both images and video and is reported to deliver metric accuracy, sharpness, and temporal consistency together, outperforming prior work on those dimensions. The released model family spans a Pareto frontier across metric accuracy, sharpness, temporal consistency, and runtime; the excerpt does not provide quantitative comparisons or details on runtime.

### Source excerpt

arXiv:2611.26603v1 Announce Type: cross Abstract: We present Depth Pro 2, an approach to estimating dense metric depth for images and video at full resolution. By analyzing prior methods for monocular depth estimation, we find that they commonly trade off temporal consistency and boundary sharpness: multi-frame methods favor consistency, single-frame methods favor sharpness. To overcome this trade-off, we train a single, scalable network that processes both images and video and predicts depth maps at native resolution. Depth Pro 2 produces depth maps that are metrically accurate, sharp, and temporally consistent all at once, outperforming prior work in all of these dimensions. We release a family of models that define a new Pareto frontier in terms of metric accuracy, sharpness, temporal consistency, and runtime. We release code and weights at https://apple-aiml-research.github.io/ml-depth-pro-2/

## Online Mistake Detection in Instruction-Conditioned Procedural Video

DevFeed: [Online Mistake Detection in Instruction-Conditioned Procedural Video](<https://devfeed.tech/articles/online-mistake-detection-in-instruction-conditioned-procedural-video-145538.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25567>)

Author: Amit Feldman, Ilan Shimshoni, Rotem Dror

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Video Analytics](<https://devfeed.tech/topics/video-analytics.md>), [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>), [image recognition](<https://devfeed.tech/topics/image-recognition.md>)

Tags: [actions](<https://devfeed.tech/tags/actions.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [detection](<https://devfeed.tech/tags/detection.md>), [error](<https://devfeed.tech/tags/error.md>), [error-propagation](<https://devfeed.tech/tags/error-propagation.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [execution](<https://devfeed.tech/tags/execution.md>), [f1](<https://devfeed.tech/tags/f1.md>), [instructions](<https://devfeed.tech/tags/instructions.md>), [localization](<https://devfeed.tech/tags/localization.md>), [mistakes](<https://devfeed.tech/tags/mistakes.md>), [model](<https://devfeed.tech/tags/model.md>), [modular](<https://devfeed.tech/tags/modular.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [offline](<https://devfeed.tech/tags/offline.md>), [online](<https://devfeed.tech/tags/online.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [predictions](<https://devfeed.tech/tags/predictions.md>), [procedural](<https://devfeed.tech/tags/procedural.md>), [protocol](<https://devfeed.tech/tags/protocol.md>), [video](<https://devfeed.tech/tags/video.md>)

### AI overview

The paper studies online mistake detection in egocentric procedural video, conditioning predictions on task instructions, observed video and prior predictions. The system identifies and temporally localizes completed action attempts, then classifies them as correct, modified, slips, corrections or additions; omissions are assessed after the video ends. It compares a modular pipeline for step identification, localization and execution verification with a joint model predicting all outputs at once. After repartitioning EgoPER to include erroneous videos in every split, both approaches achieve similar step-segmentation F1@0.5 (93.6% modular; 92.1% joint). The joint model scores higher on binary mistake detection (84.1% vs. 80.7%), especially for slips (56.4% vs. 37.5%). Component analysis suggests stage-to-stage error propagation contributes to the modular model's lower detection score. On test videos containing mistakes, both approaches outperform released AEM and ESTANet checkpoints under the paper's evaluation protocol.

### Source excerpt

arXiv:2611.25567v1 Announce Type: cross Abstract: An assistant monitoring a person performing a procedure is most useful when it can detect mistakes while there is still time to recover. This requires judging actions against task instructions as the video unfolds, identifying the attempted step, locating it in time, and assessing its execution. Existing online detectors do not use the instruction text, while instruction-conditioned methods operate offline. In this paper, we study online, instruction-conditioned mistake detection in egocentric videos. Using only observed video, task instructions, and prior predictions, our model identifies and localizes completed action attempts and classifies them as correct, modified, slip, corrections, or additions; omitted steps are determined after the video ends. We compare a modular pipeline, with separate step identification, temporal localization, and execution verification, to a joint model that predicts all outputs in a single step. To support supervised learning from mistakes, we repartition EgoPER to include erroneous videos in all splits. The approaches achieve similar step-segmentation F1@0.5 (93.6% and 92.1%), while the joint model achieves higher binary mistake-detection F1@0.5 (84.1% vs. 80.7%), with its largest gain on slips (56.4% vs. 37.5%). Component-level analysis suggests that error propagation between stages contributes to the modular approach's lower detection score. On test videos containing mistakes, both designs outperform released AEM and ESTANet checkpoints under our evaluation protocol.

## Kandinsky 6.0 Video: Foundation Models for Synchronized Video and Audio Generation

DevFeed: [Kandinsky 6.0 Video: Foundation Models for Synchronized Video and Audio Generation](<https://devfeed.tech/articles/kandinsky-6-0-video-foundation-models-for-synchronized-video-and-audio-generation-144957.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.05608>)

Author: Team Kandinsky, Julia Agafonova, Bulat Akhmatov, Mikhail Aksyutin, Grigorii Alekseenko, Anastasia Aliaskina, Olga Androsova, Vladimir Arkhipkin, Anna Averchenkova, Alexander Belykh, Serafima Bocharova

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>), [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>), [Generative AI](<https://devfeed.tech/topics/generative-ai.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [diffusers](<https://devfeed.tech/topics/diffusers.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [attention](<https://devfeed.tech/tags/attention.md>), [audio](<https://devfeed.tech/tags/audio.md>), [audio-video](<https://devfeed.tech/tags/audio-video.md>), [code](<https://devfeed.tech/tags/code.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-mm](<https://devfeed.tech/tags/cs-mm.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [diffusion-models](<https://devfeed.tech/tags/diffusion-models.md>), [distillation](<https://devfeed.tech/tags/distillation.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [foundation](<https://devfeed.tech/tags/foundation.md>), [foundation-models](<https://devfeed.tech/tags/foundation-models.md>), [generate](<https://devfeed.tech/tags/generate.md>), [generation](<https://devfeed.tech/tags/generation.md>), [image](<https://devfeed.tech/tags/image.md>), [integration](<https://devfeed.tech/tags/integration.md>), [kandinsky](<https://devfeed.tech/tags/kandinsky.md>), [kandinsky-6-0-video](<https://devfeed.tech/tags/kandinsky-6-0-video.md>), [models](<https://devfeed.tech/tags/models.md>), [video](<https://devfeed.tech/tags/video.md>)

### AI overview

Kandinsky 6.0 Video offers 3B-parameter Lite and 29B-parameter Pro diffusion models for text- or image-conditioned generation of five-second clips with synchronized 44 kHz audio and lip-sync; built-in super-resolution targets Full HD. Its dual-stream CrossDiT links a pretrained video stream and a newly trained audio stream with bidirectional cross-attention. Training progresses from audio-only pretraining to joint paired audio-video training, followed by supervised fine-tuning, reinforcement-learning post-training, and distillation. In side-by-side human evaluation, Pro outperformed Kandinsky 5.0 Video Pro and was competitive with leading models; the authors report better speech quality than LTX 2.5 and Seedance 2.0. Code, checkpoints, and diffusers integration are released under the MIT license.

### Source excerpt

arXiv:2610.05608v2 Announce Type: replace-cross Abstract: We present Kandinsky 6.0 Video, a family of foundation diffusion models for synchronized text-to-audio-video generation, comprising Kandinsky 6.0 Video Lite (3B parameters) and Kandinsky 6.0 Video Pro (29B parameters). Both models generate 5-second video clips with synchronized 44 kHz audio, including lip-sync, in text-to-audio-video (T2AV) and image-to-audio-video (I2AV) modes; a built-in super-resolution model raises the output resolution to Full-HD (1920x1080). Building on the video generation capabilities of Kandinsky 5.0, Kandinsky 6.0 Video employs a dual-stream CrossDiT architecture that connects a pretrained video stream and a newly trained audio stream through bidirectional cross-attention for temporal and semantic alignment. Our continuous pretraining strategy first trains the audio stream from scratch on large-scale audio corpora and then trains both streams jointly on paired audio-video data while preserving unimodal fidelity; pretraining is followed by supervised fine-tuning, reinforcement-learning-based post-training, and distillation. In side-by-side human evaluation, Kandinsky 6.0 Video Pro clearly outperforms its predecessor, Kandinsky 5.0 Video Pro, and remains competitive with leading audio-video generation models, outperforming LTX 2.5 and Seedance 2.0 in speech quality. To accelerate open research and deployment in multimedia generation, we release the code, model checkpoints, and diffusers integration under the MIT license.

## Text-Aware, Vision-Blind: How Multimodal Image Editing Systems Handle Unsafe Inputs

DevFeed: [Text-Aware, Vision-Blind: How Multimodal Image Editing Systems Handle Unsafe Inputs](<https://devfeed.tech/articles/text-aware-vision-blind-how-multimodal-image-editing-systems-handle-unsafe-inputs-145194.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26569>)

Author: Ziqing Yang, Yixin Wu, Michael Backes, Yang Zhang

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [image editing](<https://devfeed.tech/topics/image-editing.md>), [ai evaluation](<https://devfeed.tech/topics/ai-evaluation.md>), [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>), [ai security](<https://devfeed.tech/topics/ai-security.md>), [qwen](<https://devfeed.tech/topics/qwen.md>)

Tags: [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [image](<https://devfeed.tech/tags/image.md>), [image-editing](<https://devfeed.tech/tags/image-editing.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [system](<https://devfeed.tech/tags/system.md>)

### AI overview

EditSafetyBench evaluates safety in image editing using 5,940 image-instruction pairs spanning 11 risk categories and three edit intents. Across Gemini, GPT, Qwen API, and Qwen Local, the authors report a mismatch between text and image handling: Gemini and GPT refuse 87.1% and 81.6% of unsafe-intent prompts, but only 24.2% and 22.4% of cases pairing unsafe images with neutral instructions. Results also vary by category; for unsafe political content, GPT allows 70.4% of editing attempts versus Gemini's 34.8%. Sexual content shows the strongest visual sensitivity. The authors argue that editing safety should jointly assess image content and text prompts.

### Source excerpt

arXiv:2611.26569v1 Announce Type: cross Abstract: Multimodal image generation systems have made rapid progress and increasingly emphasize content safety. However, their safety mechanisms tend to rely on a single modality and lack joint consideration of both visual content and text prompts, particularly in instruction-guided image editing, where the uploaded image may itself carry risk. We thus introduce EditSafetyBench, a benchmark that measures system-level safety awareness of advanced image editing systems, containing 5,940 image-instruction pairs, covering safe and unsafe images across 11 risk categories and three edit intents. We then probe the benchmark dataset on four systems, including Gemini, GPT, Qwen API, and Qwen Local. Based on our evaluation, we observe a systematic text-vision safety mismatch: Gemini and GPT refuse 87.1\% and 81.6\% cases containing unsafe-intent editing prompts, while only refusing 24.2\% and 22.4\% cases with both unsafe images and neutral instructions. At the category level, most risks are considered inconsistently by different systems, e.g., GPT allows 70.4\% editing attempts of unsafe political content while Gemini only allows 34.8\%. Notably, sexual content shows the strongest visual sensitivity, likely benefiting from mature NSFW identification works. Our findings highlight the need for safety guardrails that jointly reason over both visual and textual modalities in image editing systems.

## Apache Iceberg Integrates FILE Type for Images and Audio Into Next Specification: Gap Remains Between Google's Announcement and Implementation Status

DevFeed: [Apache Iceberg Integrates FILE Type for Images and Audio Into Next Specification: Gap Remains Between Google's Announcement and Implementation Status](<https://devfeed.tech/articles/apache-iceberg-integrates-file-type-for-images-and-audio-into-next-specification-gap-remains-between-google-s-announcement-and-implementation-status-144635.md>)

Original publisher: [Read original article](<https://joho-todai.com/en/apache-iceberg-file-type-image-audio-integration/>)

Author: Toru Misaki

Published: 2026-11-19T23:53:13Z

Content type: article

Language: en

Sources: [Joho no Todai](<https://devfeed.tech/sources/joho-no-todai.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [implementation](<https://devfeed.tech/topics/implementation.md>), [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>), [Image](<https://devfeed.tech/topics/image.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>)

Tags: [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [audio](<https://devfeed.tech/tags/audio.md>), [file](<https://devfeed.tech/tags/file.md>), [google](<https://devfeed.tech/tags/google.md>), [images](<https://devfeed.tech/tags/images.md>), [implementation](<https://devfeed.tech/tags/implementation.md>), [specification](<https://devfeed.tech/tags/specification.md>)

### AI overview

Apache Iceberg's FILE type standardizes references to images, audio, and documents as table values, with fields for uri, byte range, MIME type, checksum, and optional inline data. Subfield statistics may help engines filter data by content type. This could reduce reliance on engine-specific file handling and auxiliary metadata, but it does not guarantee that referenced files exist or manage their ownership, access, deletion, or relocation; readers must compare checksums to detect changes. The specification text was merged into Iceberg's main branch, but v4 remains under development and has not been formally adopted. An initial Java API implementation is merged; serialization, Avro/Parquet/ORC support, Spark and Flink support, and other work remain. The excerpt therefore describes a developing standard, not established cross-engine interoperability.

### Source excerpt

The Apache Iceberg FILE type, introduced by Google on November 19, describes images, audio, and documents as typed table data. The specification has been merged into the main branch, but v4 has not yet been formally adopted and processing-engine implementation remains at an early stage.

## Bringing the lake back to the lakehouse: Introducing the FILE type in Apache Iceberg

DevFeed: [Bringing the lake back to the lakehouse: Introducing the FILE type in Apache Iceberg](<https://devfeed.tech/articles/bringing-the-lake-back-to-the-lakehouse-introducing-the-file-type-in-apache-iceberg-146546.md>)

Original publisher: [Read original article](<http://opensource.googleblog.com/2026/11/iceberg-file-type-multimodal-lakehouse.html>)

Author: Google Open Source (noreply@blogger.com)

Published: 2026-11-19T19:30:00Z

Content type: article

Language: en

Sources: [Google Open Source Blog](<https://devfeed.tech/sources/google-open-source-blog.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [collective](<https://devfeed.tech/tags/collective.md>), [columnar](<https://devfeed.tech/tags/columnar.md>), [consistency](<https://devfeed.tech/tags/consistency.md>), [data](<https://devfeed.tech/tags/data.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [data-type](<https://devfeed.tech/tags/data-type.md>), [data-warehouse](<https://devfeed.tech/tags/data-warehouse.md>), [external-storage](<https://devfeed.tech/tags/external-storage.md>), [file](<https://devfeed.tech/tags/file.md>), [files](<https://devfeed.tech/tags/files.md>), [formats](<https://devfeed.tech/tags/formats.md>), [google-cloud](<https://devfeed.tech/tags/google-cloud.md>), [guarantees](<https://devfeed.tech/tags/guarantees.md>), [isolation](<https://devfeed.tech/tags/isolation.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [object-storage](<https://devfeed.tech/tags/object-storage.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [organization](<https://devfeed.tech/tags/organization.md>), [type](<https://devfeed.tech/tags/type.md>)

### AI overview

Apache Iceberg's new `FILE` type models external assets as structured values containing fields such as `uri`, `offset`, `size`, `content_type`, `checksum`, and optional inline bytes, addressing limitations of bare string paths. The excerpt says this lets supporting engines reuse stored metadata instead of fetching it per file, use MIME types rather than guessing from extensions, detect possible changes through checksums, and read specified byte ranges. It also places structured records and media references in the same transactional table. The specification is described as finalized and targeted for Iceberg v4; native reader and writer support across engines including Spark, Trino, and DuckDB is presented as future work. The excerpt does not establish current implementation availability.

### Source excerpt

by Sung Yun & Gaurav Soni, Google Cloud The original promise of the data lake was simple: a single, universal home for all an organization's data. Whether structured database exports, raw text documents, audio recordings, or high-resolution imagery, everything landed in object storage. But as data lakes grew into sprawling swamps lacking transactional guarantees and consistency, the industry pioneered the lakehouse. Open table formats like Apache Iceberg brought order to the chaos, introducing data warehouse rigor--ACID transactions, snapshot isolation, partition evolution, and fast columnar pruning--directly to cloud storage. Yet, in our collective rush to build the "house", we quietly left the "lake" behind. Photo by mydaydream89 (CC0) via Wikimedia Commons With the rapid rise of multimodal Large Language Models (LLMs), interest in processing unstructured data directly within the lakehouse has grown rapidly. Multimodal LLMs enable modern data warehouses and lakehouses to analyze unstructured inputs--such as raw images, audio, and documents--much like traditional structured data, without requiring complex, bespoke machine learning staging pipelines. Because table formats were designed around structured rows and columns, unstructured media was left behind. Raw files were relegated back to external storage buckets, loosely connected to analytical tables through plain, unvalidated string URIs. The Apache Iceberg community recently took a major step toward reuniting these two worlds by merging support for the new FILE data type into the Iceberg specification. Limitations of using bare string URIs Historically, platforms referenced external unstructured assets by storing simple string paths inside tabular columns. While straightforward, relying on plain string URIs introduces several operational challenges: No file metadata in the table: A string path doesn't say what type a file is or how big it is. Engines guess types from file extensions, or make a separate call to objec

## Gaussian Primitive Fusion for Any-Resolution Multimodal Images

DevFeed: [Gaussian Primitive Fusion for Any-Resolution Multimodal Images](<https://devfeed.tech/articles/gaussian-primitive-fusion-for-any-resolution-multimodal-images-142761.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24164>)

Author: Baoshuo Cai, Xinrui Jiang, Bing Cao

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Image processing](<https://devfeed.tech/topics/image-processing.md>), [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>), [Image](<https://devfeed.tech/topics/image.md>)

Tags: [artifacts](<https://devfeed.tech/tags/artifacts.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [continuous](<https://devfeed.tech/tags/continuous.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [fusion](<https://devfeed.tech/tags/fusion.md>), [gaussian](<https://devfeed.tech/tags/gaussian.md>), [images](<https://devfeed.tech/tags/images.md>), [infrared](<https://devfeed.tech/tags/infrared.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [resolutions](<https://devfeed.tech/tags/resolutions.md>), [super-resolution](<https://devfeed.tech/tags/super-resolution.md>), [textures](<https://devfeed.tech/tags/textures.md>)

### AI overview

Gaussian Primitive Fusion (GPF) combines multimodal images at differing native resolutions in a shared continuous Gaussian space, avoiding the initial resampling used by common-grid fusion. It fits native-resolution observations with shared Gaussian geometry and modality-specific content, learns their contributions in that representation, then renders the fused result at the finest native resolution. The abstract reports clearer contours and finer textures with fewer blurring and blocking artifacts than resample-then-fuse pipelines, and the best quantitative results across multiple infrared-visible benchmarks compared with state-of-the-art methods using interpolation or super-resolution frontends.

### Source excerpt

arXiv:2611.24164v1 Announce Type: cross Abstract: Multimodal image fusion integrates complementary observations from different modalities into a unified representation. Existing methods typically estimate modality contributions on a common discrete grid. When source modalities have different native resolutions, constructing such a grid generally involves resampling, which may introduce artifacts and affect subsequent fusion decisions. To address this problem, we propose Gaussian Primitive Fusion (GPF), which reformulates heterogeneous-resolution fusion in a shared continuous Gaussian space. Instead of first converting source images to a common resolution, GPF directly fits their native-resolution observations with shared Gaussian geometry and modality-specific content representations, and learns modality contributions within the resulting continuous representation. Once optimized, the fused representation is rendered at the finest native resolution without resampling the sources. With heterogeneous-resolution inputs, GPF produces clearer target contours and finer textures, with less blurring and blocking artifacts than resample-then-fuse pipelines. Quantitatively, it achieves the best results on multiple infrared-visible benchmarks against state-of-the-art fusion methods equipped with interpolation or super-resolution frontends.

## Rethinking Contrastive Targets in Radiology Language-Image Pretraining

DevFeed: [Rethinking Contrastive Targets in Radiology Language-Image Pretraining](<https://devfeed.tech/articles/rethinking-contrastive-targets-in-radiology-language-image-pretraining-143472.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24016>)

Author: Yunhe Gao, Ashwin Kumar, Jiaming Liu, Chong Wang, Maya Varma, Fuying Wang, Stefania L. Moroianu, Louisa Fay, Sergios Gatidis, Akshay S. Chaudhari, Curtis Langlotz

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>)

Tags: [3d](<https://devfeed.tech/tags/3d.md>), [batch](<https://devfeed.tech/tags/batch.md>), [batches](<https://devfeed.tech/tags/batches.md>), [bias](<https://devfeed.tech/tags/bias.md>), [compatibility](<https://devfeed.tech/tags/compatibility.md>), [contrastive-learning](<https://devfeed.tech/tags/contrastive-learning.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [failures](<https://devfeed.tech/tags/failures.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [image](<https://devfeed.tech/tags/image.md>), [language](<https://devfeed.tech/tags/language.md>), [learning-theory](<https://devfeed.tech/tags/learning-theory.md>), [pretraining](<https://devfeed.tech/tags/pretraining.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>), [targets](<https://devfeed.tech/tags/targets.md>), [testbed](<https://devfeed.tech/tags/testbed.md>), [training](<https://devfeed.tech/tags/training.md>), [zero-shot](<https://devfeed.tech/tags/zero-shot.md>)

### AI overview

CLIP-style radiology pretraining can show worsening zero-shot performance even as training loss falls, and can fail to benefit from large batches. The abstract attributes both failures to a biased paired-identity target: unpaired reports may share clinical findings with an image, while the training recipe treats them as negatives. A synthetic testbed reproduces the failures, and the proposed theory links them to deterministic gradient bias amplified by batch size. CorLIP treats text as positive when its clinical assertions agree with the image, without changing the architecture. The abstract reports less post-peak degradation, successful large-batch training (stable up to B=2048), stronger gains from more data, and transfer from chest X-ray to 3D CT.

### Source excerpt

arXiv:2611.24016v1 Announce Type: cross Abstract: CLIP-style contrastive image-text pretraining is widely used in radiology data, where each image is paired with one report and every other report in the batch is treated as a negative. Under this recipe, two failures appear, neither of which is systematically addressed by existing contrastive learning theory: zero-shot evaluation peaks then degrades while training loss keeps decreasing, and training fails to learn from large batches. We hypothesize the cause is structural rather than radiology-specific: many unpaired reports share clinical findings with a given image, so CLIP's paired-identity target is systematically biased, not merely noisy. To test this, we construct CLEVR-Radiology, a synthetic testbed that preserves only this overlapping-fact structure and reproduces both failures. With evidence that the structure is sufficient to reproduce the failures, we develop a target-misspecification theory that explains both failures from a single deterministic gradient bias amplified by batch size, and use it to motivate a minimal compatibility-based correction, CorLIP: text is treated as positive whenever its clinical assertions agree with the image, with no architecture changes. Beyond benchmark improvements, CorLIP largely removes the post-peak degradation, makes large-batch training succeed where it previously collapsed, substantially amplifies the gains from more data, and is the only configuration that remains stable up to B=2048. The correction transfers from chest X-ray to 3D CT modalities.

## Searching the Pareto Front: Multi-Objective Retrieval for Heterogeneous Scientific Modalities

DevFeed: [Searching the Pareto Front: Multi-Objective Retrieval for Heterogeneous Scientific Modalities](<https://devfeed.tech/articles/searching-the-pareto-front-multi-objective-retrieval-for-heterogeneous-scientific-modalities-142684.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23639>)

Author: Song Young Oh, Nesar Ramachandra, Bin Xia, Ian Foster, Kyle Chard

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [AI-driven information retrieval](<https://devfeed.tech/topics/ai-driven-information-retrieval.md>), [Embeddings](<https://devfeed.tech/topics/embeddings.md>), [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>), [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [approach](<https://devfeed.tech/tags/approach.md>), [astro-ph-im](<https://devfeed.tech/tags/astro-ph-im.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [embeddings](<https://devfeed.tech/tags/embeddings.md>), [fm](<https://devfeed.tech/tags/fm.md>), [foundation-models](<https://devfeed.tech/tags/foundation-models.md>), [fusion](<https://devfeed.tech/tags/fusion.md>), [galaxy](<https://devfeed.tech/tags/galaxy.md>), [history](<https://devfeed.tech/tags/history.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [nearest-neighbor](<https://devfeed.tech/tags/nearest-neighbor.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [pareto-optimality](<https://devfeed.tech/tags/pareto-optimality.md>), [properties](<https://devfeed.tech/tags/properties.md>), [proximity](<https://devfeed.tech/tags/proximity.md>), [query](<https://devfeed.tech/tags/query.md>), [raw](<https://devfeed.tech/tags/raw.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [searching](<https://devfeed.tech/tags/searching.md>), [similarity](<https://devfeed.tech/tags/similarity.md>)

### AI overview

The study compares a multimodal foundation model (FM) with separately trained modality embeddings combined through Pareto-based multi-objective optimization for galaxy retrieval across star-formation histories, spectra, and observations from two telescopes. The FM provides strong embedding-space proximity, while late fusion retains modality-specific distances so scientists can inspect which physical signals contribute to similarity. Across six measured properties, the method's nearest neighbor is reported to be 8% closer to the query in raw-data space on average than kNN in the FM embedding space (spread 0.128 versus 0.139). The abstract suggests this may surface astrophysically coherent candidates overlooked by single-objective search.

### Source excerpt

arXiv:2611.23639v1 Announce Type: new Abstract: In cosmology, surveys span different sky regions and wavelength ranges, resulting in uneven modality coverage across observational datasets. Such heterogeneity can be addressed through either multimodal foundation models (FMs), which learn shared embedding spaces, or independently learned modality embeddings combined via late fusion. We investigate whether the latter provides an adaptive alternative to FM-based scientific retrieval. We evaluate this approach on a galaxy retrieval task spanning diverse modalities--history of star formation, spectra of light emitted, and multi-wavelength observations from two telescopes--comparing an FM baseline with separately trained embeddings fused through multi-objective optimization based on Pareto optimality. We find that the FM baseline achieves strong embedding-space proximity, but our method preserves modality-specific distance structure that allows scientists to inspect which physical signals drive similarity. Our method also achieves improved fidelity in raw-data space: across six measured properties, its nearest neighbor is 8% closer to the query on average than that retrieved by kNN in the FM embedding space (spread 0.128 vs. 0.139). These results suggest that multi-objective retrieval can surface astrophysically coherent candidates that single-objective search alone may overlook.

## H3Remover: Reinforcing Video Object Removal without Paired Data

DevFeed: [H3Remover: Reinforcing Video Object Removal without Paired Data](<https://devfeed.tech/articles/h3remover-reinforcing-video-object-removal-without-paired-data-142815.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24633>)

Author: Kangjie Chen, Xiangyu Li, Jinhao Deng, Dongbin Zhang, Chaoda Zheng, Shijia Chen, Yuqi Zhang, Choo Sin Wai, Boyi Sun, Puyi Wang, Yu Zhang, Xianming Liu, Boyang Wang

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Generative AI](<https://devfeed.tech/topics/generative-ai.md>), [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>)

Tags: [adapters](<https://devfeed.tech/tags/adapters.md>), [appearance](<https://devfeed.tech/tags/appearance.md>), [background](<https://devfeed.tech/tags/background.md>), [comparisons](<https://devfeed.tech/tags/comparisons.md>), [completion](<https://devfeed.tech/tags/completion.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [data](<https://devfeed.tech/tags/data.md>), [editors](<https://devfeed.tech/tags/editors.md>), [edits](<https://devfeed.tech/tags/edits.md>), [effects](<https://devfeed.tech/tags/effects.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [fixed](<https://devfeed.tech/tags/fixed.md>), [lora](<https://devfeed.tech/tags/lora.md>), [minimax](<https://devfeed.tech/tags/minimax.md>), [minimax-h3](<https://devfeed.tech/tags/minimax-h3.md>), [object](<https://devfeed.tech/tags/object.md>), [ordering](<https://devfeed.tech/tags/ordering.md>), [original](<https://devfeed.tech/tags/original.md>), [video](<https://devfeed.tech/tags/video.md>)

### AI overview

H3Remover reinforces MiniMax-H3 Base Ref2VA for video object removal without paired object-free training targets. It combines a depth-based removal reward, a background-geometry reward that preserves relative depth ordering, and an appearance reward that limits RGB changes in protected regions while exempting targets and estimated shadows or reflections. Frozen HPSv3 provides perceptual-quality feedback; DiffusionART optimizes LoRA adapters with the backbone frozen and evaluation regions fixed across candidates. The excerpt reports substantial reconstruction gains over original H3 and qualitative improvements in removal and surrounding-scene preservation.

### Source excerpt

arXiv:2611.24633v1 Announce Type: cross Abstract: Pretrained video editors possess rich removal and completion priors, yet may retain targets or alter unrelated content. We present H3Remover, which reinforces MiniMax-H3 Base Ref2VA for video object removal without paired object-free training targets. To discourage superficial edits such as recoloring, a target-removal reward favors calibrated depth changes consistent with revealing farther content. To limit unintended scene changes, a background-geometry reward encourages preservation of relative depth ordering. To preserve background appearance without retaining unwanted shadows or reflections, an appearance reward constrains RGB changes in source-derived protected regions while exempting targets and estimated effects. Frozen HPSv3 complements these constraints with perceptual-quality feedback for synthesized content. Evaluation regions remain fixed across candidates, and DiffusionART optimizes LoRA adapters with the backbone frozen. Experiments show substantial reconstruction gains over original H3, with qualitative comparisons demonstrating more effective object removal and better preservation of surrounding scene content.

## SIGMA: speaker-isolated, symptom-structured multimodal estimation of depression severity from clinical interviews

DevFeed: [SIGMA: speaker-isolated, symptom-structured multimodal estimation of depression severity from clinical interviews](<https://devfeed.tech/articles/sigma-speaker-isolated-symptom-structured-multimodal-estimation-of-depression-severity-from-clinical-interviews-142642.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23348>)

Author: Zhuoyuan Tang, Tao Wang, Kai Yang, Li Yuan, Yuezhou Zhang, Rongrong Zhong, Yi Cai, Saskia C. Sanderson, Jaime Delgadillo, Richard Dobson, Angus Roberts

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>)

Tags: [assessment](<https://devfeed.tech/tags/assessment.md>), [attention](<https://devfeed.tech/tags/attention.md>), [audio](<https://devfeed.tech/tags/audio.md>), [automated](<https://devfeed.tech/tags/automated.md>), [cross](<https://devfeed.tech/tags/cross.md>), [cross-attention](<https://devfeed.tech/tags/cross-attention.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-sd](<https://devfeed.tech/tags/cs-sd.md>), [depression](<https://devfeed.tech/tags/depression.md>), [diarization](<https://devfeed.tech/tags/diarization.md>), [interviews](<https://devfeed.tech/tags/interviews.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [performance](<https://devfeed.tech/tags/performance.md>), [sigma](<https://devfeed.tech/tags/sigma.md>), [speaker](<https://devfeed.tech/tags/speaker.md>), [speaker-diarization](<https://devfeed.tech/tags/speaker-diarization.md>), [speech](<https://devfeed.tech/tags/speech.md>), [structured](<https://devfeed.tech/tags/structured.md>), [text](<https://devfeed.tech/tags/text.md>), [video](<https://devfeed.tech/tags/video.md>)

### AI overview

SIGMA estimates depression severity from semi-clinical interviews by diarizing speakers and using only participant speech, jointly conditioning text, audio, and video through cross-attention, and predicting eight questionnaire items together via a shared latent severity trait with item-specific ordered thresholds. On the Extended Distress Analysis Interview Corpus, it reports CCC 0.762, RMSE 4.13, and MAE 3.30 across five seeds. Ablations indicate that participant-speech isolation and structured joint symptom prediction each substantially improve concordance and accuracy. These results are reported for the described corpus and setup.

### Source excerpt

arXiv:2611.23348v1 Announce Type: cross Abstract: Clinical interviews are central to depression assessment but time-consuming and rater-dependent. Automated severity estimation offers scalable screening, yet existing methods often retain interviewer speech (confounding participant cues), fuse modalities late (losing fine-grained cross-modal dynamics), or score symptoms independently (ignoring inter-symptom correlations). We present SIGMA, a speaker-aware, symptom-structured framework for depression-severity estimation from semi-clinical interviews. SIGMA uses speaker diarization to restrict input to participant speech, cross-attention so that text, audio and video condition one another during encoding, and a graded-response head that predicts the eight questionnaire items jointly through a shared latent severity trait with item-specific ordered thresholds. On the Extended Distress Analysis Interview Corpus, SIGMA achieves state-of-the-art performance (CCC 0.762, RMSE 4.13, MAE 3.30 over five seeds). Ablations confirm that isolating participant speech and joint, structured symptom prediction each substantially improve concordance and estimation accuracy.

## Risk-Controlled Correction for Frozen Multimodal Models under Missing Modalities

DevFeed: [Risk-Controlled Correction for Frozen Multimodal Models under Missing Modalities](<https://devfeed.tech/articles/risk-controlled-correction-for-frozen-multimodal-models-under-missing-modalities-142717.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23886>)

Author: Hoang Pham Cong, Long D. Nguyen, Ho Viet Duc Luong, Nguyen Tran Minh Nhat, Binh P. Nguyen, Huynh Thi Thanh Binh

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>)

Tags: [bounded](<https://devfeed.tech/tags/bounded.md>), [conditional](<https://devfeed.tech/tags/conditional.md>), [cross-entropy](<https://devfeed.tech/tags/cross-entropy.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [gap](<https://devfeed.tech/tags/gap.md>), [headroom](<https://devfeed.tech/tags/headroom.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [multimodal-models](<https://devfeed.tech/tags/multimodal-models.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [risk](<https://devfeed.tech/tags/risk.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>)

### AI overview

For frozen multimodal predictors facing missing inputs, observed error is correctable only to the extent that the prediction differs from the Bayes prediction given the remaining modalities; ambiguity in those modalities remains. The abstract says excess risk captures this gap for cross-entropy and squared error. GUARD leaves the deployed predictor unchanged, uses retrieval to propose a correction, learns from deployment-matched labeled data when to apply it, and sets a threshold on a held-out split through conformal risk control. Under exchangeability, the marginal chance of applying a correction whose loss increase exceeds a chosen tolerance is bounded by a preset level, regardless of the learned score. A conditionally correct model with more modalities can supply the Bayes target by averaging outputs conditional on the remaining modalities without retrieval labels; without such an assumption, unlabeled data alone cannot identify the target. Across nine frozen model-dataset pairs in six domains, GUARD reportedly retained most blanket correction gains while staying within harm budgets that blanket correction often exceeded.

### Source excerpt

arXiv:2611.23886v1 Announce Type: new Abstract: A deployed multimodal predictor may face missing modalities after its weights can no longer be changed. This raises a question that training-time repair does not answer: what part of the resulting error is still correctable from the information that remains? We show that this part is characterized by the Bayes gap, the difference between the model's prediction and the Bayes prediction given the observed modalities. For cross-entropy and squared error, excess risk measures this gap in the geometry of the loss; ambiguity in the observed modalities cannot be removed. Thus, a badly degraded prediction is not necessarily a correctable one. Estimating the Bayes prediction normally requires labels. However, when a model observing more modalities is conditionally correct, averaging its outputs conditional on the remaining modalities recovers this target without retrieval labels; with the same fixed blend and neighbors, it is no worse in expected cross-entropy or Brier risk than averaging hard labels. Some such assumption is unavoidable: otherwise the target is not identifiable from unlabeled data alone. GUARD (Gated Utility-Aware Recalibration for Deployment) leaves the deployed predictor unchanged. It estimates a candidate correction by retrieval, learns from deployment-matched labeled data when correction is likely to help, and calibrates a threshold on a held-out split by conformal risk control. Under exchangeability, the marginal probability that GUARD applies a correction whose loss increase exceeds a prescribed tolerance is bounded by a preset level, regardless of the learned score. In controlled experiments, correction gain follows headroom rather than raw error; across nine frozen model-dataset pairs spanning six domains, GUARD retains most of blanket correction's gain within harm budgets that blanket correction often exceeds.

## Liquid AI's d1.2 and d1.2-ultra are now available on AI Gateway

DevFeed: [Liquid AI's d1.2 and d1.2-ultra are now available on AI Gateway](<https://devfeed.tech/articles/liquid-ai-s-d1-2-and-d1-2-ultra-are-now-available-on-ai-gateway-144058.md>)

Original publisher: [Read original article](<https://vercel.com/changelog/liquid-ais-d1-2-and-d1-2-ultra-are-now-available-on-ai-gateway>)

Author: Zachary Chen

Published: 2026-11-19T00:00:00Z

Content type: release

Language: en

Sources: [Vercel News](<https://devfeed.tech/sources/vercel-news.md>)

Topics: [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-gateway](<https://devfeed.tech/tags/ai-gateway.md>), [ai-sdk](<https://devfeed.tech/tags/ai-sdk.md>), [api](<https://devfeed.tech/tags/api.md>), [audio](<https://devfeed.tech/tags/audio.md>), [boolean](<https://devfeed.tech/tags/boolean.md>), [box](<https://devfeed.tech/tags/box.md>), [choice](<https://devfeed.tech/tags/choice.md>), [classification](<https://devfeed.tech/tags/classification.md>), [d1](<https://devfeed.tech/tags/d1.md>), [decision](<https://devfeed.tech/tags/decision.md>), [decision-model](<https://devfeed.tech/tags/decision-model.md>), [decision-models](<https://devfeed.tech/tags/decision-models.md>), [decisions-api](<https://devfeed.tech/tags/decisions-api.md>), [documentation](<https://devfeed.tech/tags/documentation.md>), [example](<https://devfeed.tech/tags/example.md>), [examples](<https://devfeed.tech/tags/examples.md>), [faster](<https://devfeed.tech/tags/faster.md>), [liquid](<https://devfeed.tech/tags/liquid.md>), [liquid-ai](<https://devfeed.tech/tags/liquid-ai.md>)

### AI overview

Liquid AI's d1.2 and d1.2-ultra are available through AI Gateway for structured decisions, including Boolean probabilities, category choices, rubric scores, and image bounding boxes. Multiple questions can use one input; the excerpt gives examples for classifying an audio voicemail and locating an object in an image. Box coordinates are fractions of image height and width. d1.2 is described as faster and lower-cost, while d1.2-ultra is described as the most accurate. Examples require an AI Gateway API key; the AI SDK voicemail example requires `ai` 7.0.136 or later. Requests appear in AI Gateway logs and count toward budgets.

### Source excerpt

Liquid AI's d1.2 and d1.2-ultra are now available on AI Gateway with audio input and bounding box support. These decision models return structured answers instead of generating text. Ask Boolean questions for yes/no probabilities, Choice questions to select a category, or Score questions to rate an input against a rubric. Multiple questions can share the same input in one request. d1.2 is the faster, lower-cost model. d1.2-ultra is Liquid AI's most accurate decision model. Use liquid/d1.2 or liquid/d1.2-ultra as the model name. Before running the examples, set AI_GATEWAY_API_KEY to an AI Gateway API key. Classify a voicemail with AI SDK 7 Pass audio as a file part in state. This example reads voicemail.flac and returns the probability that the caller is requesting a refund. It requires ai 7.0.136 or later. For image classification, pass image bytes with the matching media type, such as image/png. Locate an object with the Decisions API Box questions return top and bottom as fractions of the image height, and left and right as fractions of its width. This Python example uses the Decisions API to locate a signature in form.png and print its bounding box: Decision requests appear in AI Gateway logs and count toward budgets. Learn more in the decision model documentation, or browse all decision models. Read more

## The Illusion of Grounding: Do Vision-Language Models Weigh Retrieved Text and Images Like an Ideal Observer?

DevFeed: [The Illusion of Grounding: Do Vision-Language Models Weigh Retrieved Text and Images Like an Ideal Observer?](<https://devfeed.tech/articles/the-illusion-of-grounding-do-vision-language-models-weigh-retrieved-text-and-images-like-an-ideal-observer-141848.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23127>)

Author: Basavaraj R M, Harshal Chandel

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [vlm](<https://devfeed.tech/topics/vlm.md>), [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>), [Vectors & LLMs](<https://devfeed.tech/topics/vectors-llms.md>)

Tags: [audit](<https://devfeed.tech/tags/audit.md>), [bayesian](<https://devfeed.tech/tags/bayesian.md>), [bias](<https://devfeed.tech/tags/bias.md>), [capture](<https://devfeed.tech/tags/capture.md>), [coco](<https://devfeed.tech/tags/coco.md>), [code](<https://devfeed.tech/tags/code.md>), [consistency](<https://devfeed.tech/tags/consistency.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cue](<https://devfeed.tech/tags/cue.md>), [error](<https://devfeed.tech/tags/error.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [gaussian](<https://devfeed.tech/tags/gaussian.md>), [grounding](<https://devfeed.tech/tags/grounding.md>), [images](<https://devfeed.tech/tags/images.md>), [language](<https://devfeed.tech/tags/language.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [models](<https://devfeed.tech/tags/models.md>), [text](<https://devfeed.tech/tags/text.md>), [vision](<https://devfeed.tech/tags/vision.md>), [vision-language-models](<https://devfeed.tech/tags/vision-language-models.md>)

### AI overview

On clean inputs, four evaluated open vision-language models achieve 92.5%-94.2% accuracy, but under retrieval or image stress accuracy drops to 41.3%-59.9%. The study introduces GIB-200K, a 200,000-item benchmark built from COCO and Visual Genome with controlled retrieval noise, visual degradation, and cross-modal conflicts, and methods to estimate modality weighting and measure answers not grounded in images. The two LLaVA models overweight retrieved text, while InternVL3 favors image evidence; 34.8%-71.4% of clean-correct answers are abandoned when documents are misleading. A training-free check comparing answers with and without retrieval trades coverage for accuracy under misleading documents, while image-only fallback fails when images are heavily degraded. Results cover one seeded 2,000-item split.

### Source excerpt

arXiv:2611.23127v1 Announce Type: new Abstract: Retrieval-augmented vision-language models (VLMs) answer from two sources of evidence that can each fail: the retrieved text can be wrong or noisy, and the image can be degraded or contradicted. We introduce GIB-200K, 200,000 image-question items built from COCO and Visual Genome, each expanded into controlled stress conditions: hard-negative retrieval noise (1-20 documents), visual ambiguity (Gaussian and motion blur, low-resource crops) and cross-modal conflict with counterfactual flip pairs. Instead of only measuring degradation, we ask how models combine the two modalities. From ideal Bayesian cue combination we derive a testable prediction that yields formal definitions of Knowledge Inertia (text over-weighted relative to the model's own image evidence) and Sensory Capture (the reverse), and we estimate the weights with a measurement-error-corrected regression whose recovery of known weights we verify. A counterfactual flip decomposition separates modality bias from concept priors, and a Grounding Illusion Score measures how much clean accuracy is not grounded in the image. We evaluate 4 open models on one frozen, seeded evaluation split of 2,000 GIB-200K items. All models are accurate on clean inputs (92.5%-94.2%) but adverse accuracy falls to 41.3%-59.9%. The two LLaVA models over-weight the text (Knowledge Inertia, w_T/w_I = 1.229 and 1.322), whereas InternVL3 favours the image (w_T/w_I = 0.680); between 34.8% and 71.4% of the clean answers that are correct are abandoned when the document lies. A training-free consistency check that compares the answer with and without retrieval trades coverage for accuracy under misleading documents, while falling back to the image fails when the image is heavily degraded. All per-instance outputs, code, and an audit that recomputes every reported open-model number are released.

## FaceRiff: Data-Efficient Unified Text-to-3D Face Asset Synthesis via Geometric Conditioning

DevFeed: [FaceRiff: Data-Efficient Unified Text-to-3D Face Asset Synthesis via Geometric Conditioning](<https://devfeed.tech/articles/faceriff-data-efficient-unified-text-to-3d-face-asset-synthesis-via-geometric-conditioning-139922.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21867>)

Author: Elly Akhoundi, Mathieu Lamarre, Thibault Goyallon, Patrick Anderson, \\'Etienne Danvoye

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [3D](<https://devfeed.tech/topics/3d.md>), [peft](<https://devfeed.tech/topics/peft.md>), [stable-diffusion](<https://devfeed.tech/topics/stable-diffusion.md>), [Image processing](<https://devfeed.tech/topics/image-processing.md>), [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>)

Tags: [3d](<https://devfeed.tech/tags/3d.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [asset](<https://devfeed.tech/tags/asset.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-gr](<https://devfeed.tech/tags/cs-gr.md>), [data](<https://devfeed.tech/tags/data.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [diffusion-models](<https://devfeed.tech/tags/diffusion-models.md>), [diversity](<https://devfeed.tech/tags/diversity.md>), [fixed](<https://devfeed.tech/tags/fixed.md>), [framework](<https://devfeed.tech/tags/framework.md>), [generation](<https://devfeed.tech/tags/generation.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [image](<https://devfeed.tech/tags/image.md>), [independent](<https://devfeed.tech/tags/independent.md>), [lighting](<https://devfeed.tech/tags/lighting.md>), [lora](<https://devfeed.tech/tags/lora.md>), [maps](<https://devfeed.tech/tags/maps.md>), [synthesis](<https://devfeed.tech/tags/synthesis.md>), [text](<https://devfeed.tech/tags/text.md>)

### AI overview

FaceRiff generates text-conditioned 3D face assets--mesh, albedo, and displacement--in a shared UV space, using a pretrained SDXL model with task-specific LoRA adapters and UV normal maps for geometric conditioning. It was trained on 2,049 scans from about 350 identities on a single GPU, updating less than 3% of backbone parameters per task, and produces assets without per-instance optimization. The abstract reports greater per-prompt diversity than FaceG2E and DreamFace while maintaining prompt alignment and fixed-topology UV consistency; it also generates lighting-independent albedo for PBR relighting and pore-scale displacement. In a blind study of 45 industry professionals, perceived realism was substantially higher than DreamFace and approached that of real captures.

### Source excerpt

arXiv:2611.21867v1 Announce Type: cross Abstract: High-resolution facial datasets with consistent UV-mapped geometry, albedo, and physically based displacement are scarce, yet such assets are essential for photorealistic digital-human rendering. Existing 3D face generation methods are often slow, single-modality, or require large specialized datasets. Meanwhile, large image diffusion models encode strong facial priors but do not natively produce lighting-independent textures, pore-scale displacement, or UV-aligned geometry. We present FaceRiff, a unified, data- and parameter-efficient framework for generating complete 3D face assets (mesh, albedo, and displacement) directly from text in a shared UV space. FaceRiff adapts a pretrained SDXL backbone with task-specific LoRA adapters and uses UV normal maps as geometric conditioning: per-instance normals guide albedo and displacement synthesis, while a dataset-mean normal provides a canonical spatial anchor for mesh generation. Using only 2,049 scans from approximately 350 identities and training on a single GPU, FaceRiff updates less than 3% of the backbone parameters per task and produces render-ready 3D face assets without per-instance optimization. Compared with FaceG2E and DreamFace, FaceRiff produces greater per-prompt diversity while preserving prompt alignment and fixed-topology UV consistency, and additionally generates lighting-independent albedo suitable for PBR relighting and pore-scale displacement absent from prior text-driven baselines. A blind study with 45 industry professionals shows that FaceRiff achieves substantially higher perceived realism than DreamFace, approaching that of real captures. Overall, FaceRiff demonstrates that pretrained diffusion priors can be adapted to unified 3D face asset synthesis under limited data.

## Social Signal Bench: Evidence-Weighted Evaluation of Co-occurring Social Signals in Video and Audio

DevFeed: [Social Signal Bench: Evidence-Weighted Evaluation of Co-occurring Social Signals in Video and Audio](<https://devfeed.tech/articles/social-signal-bench-evidence-weighted-evaluation-of-co-occurring-social-signals-in-video-and-audio-141801.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22704>)

Author: Siddharth Ravi, Anders R. Bargum, Hugo Anselme, Patrizio Cugia di Sant'Orsola, Alberto Cardaci, Flora Kincso Parragi, Vincent Borowiec, Line H. Clemmensen

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>), [AI Models](<https://devfeed.tech/topics/ai-models.md>), [Dataset development](<https://devfeed.tech/topics/dataset-development.md>), [Video Analytics](<https://devfeed.tech/topics/video-analytics.md>)

Tags: [audio](<https://devfeed.tech/tags/audio.md>), [bench](<https://devfeed.tech/tags/bench.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [computer-vision](<https://devfeed.tech/tags/computer-vision.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [engagement](<https://devfeed.tech/tags/engagement.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [frontier-models](<https://devfeed.tech/tags/frontier-models.md>), [recognition](<https://devfeed.tech/tags/recognition.md>), [signal](<https://devfeed.tech/tags/signal.md>), [social](<https://devfeed.tech/tags/social.md>), [video](<https://devfeed.tech/tags/video.md>)

### AI overview

Social Signal Bench evaluates co-occurring conversational signals in 47 hours of video and audio, addressing limits of conventional multi-label F1. Its video dataset covers ten signals in 25 audiovisual hours; its audio dataset covers six signals in 22 hours. Each has a public online-conversation training set, including 10 expert-annotated hours with the remainder pseudo-labeled, and a private expert-labeled test set spanning four domains. The proposed Evidence-Weighted F1 weights reference labels by annotator evidence and gives graded credit to semantically related predictions via optimal per-clip matching; controlled perturbations show it distinguishes related from unrelated errors, unlike conventional F1. Under a shared protocol, post-trained 7B open models outperform general-purpose frontier models on video and match them on audio.

### Source excerpt

arXiv:2611.22704v1 Announce Type: cross Abstract: Recognizing social signals such as interest and hesitation requires interpreting context-dependent behaviors that may co-occur in conversation. Existing affective computing benchmarks rarely evaluate such co-occurring signals, and conventional multi-label F1 overlooks semantic relationships and annotator disagreement. We introduce Social Signal Bench, a 47-hour densely annotated benchmark for the evaluation of social signal recognition and engagement, comprising two datasets: Social Signal Bench Video (25 hours), covering ten social signals in audiovisual recordings, and Social Signal Bench Audio (22 hours), covering six signals in audio-only recordings. Each dataset pairs a public training set of online conversational video, of which 10 hours are expert-annotated and the remainder pseudo-labeled, with a private expert-labeled test set spanning four conversational domains. We also propose Evidence-Weighted F1 (EW-F1), which weights reference labels by annotator evidence and assigns graded credit to semantically related predictions through optimal per-clip matching. Controlled perturbations demonstrate that EW-F1 distinguishes semantically related errors from unrelated ones, unlike conventional F1. Under a shared evaluation protocol, 7B open models post-trained on Social Signal Bench outperform general-purpose frontier models on video and match them on audio.

## HubEgo-100K: In-the-Wild Egocentric Video of Skilled Work for Robot Learning

DevFeed: [HubEgo-100K: In-the-Wild Egocentric Video of Skilled Work for Robot Learning](<https://devfeed.tech/articles/hubego-100k-in-the-wild-egocentric-video-of-skilled-work-for-robot-learning-141703.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21938>)

Author: Javad Amirian, Armin Kiani, Lo\\"ic Schaeffer, Tim Sprecher, Mohamed Taha, Karthik Yadati

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Robot Manipulation](<https://devfeed.tech/topics/robot-manipulation.md>), [Dataset development](<https://devfeed.tech/topics/dataset-development.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>), [lerobot](<https://devfeed.tech/topics/lerobot.md>)

Tags: [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [camera](<https://devfeed.tech/tags/camera.md>), [cc](<https://devfeed.tech/tags/cc.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-ro](<https://devfeed.tech/tags/cs-ro.md>), [data](<https://devfeed.tech/tags/data.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [diversity](<https://devfeed.tech/tags/diversity.md>), [environment](<https://devfeed.tech/tags/environment.md>), [environments](<https://devfeed.tech/tags/environments.md>), [imu](<https://devfeed.tech/tags/imu.md>), [kitchen](<https://devfeed.tech/tags/kitchen.md>), [learning](<https://devfeed.tech/tags/learning.md>), [lerobot](<https://devfeed.tech/tags/lerobot.md>), [privacy](<https://devfeed.tech/tags/privacy.md>), [robot](<https://devfeed.tech/tags/robot.md>), [video](<https://devfeed.tech/tags/video.md>), [work](<https://devfeed.tech/tags/work.md>), [workplace](<https://devfeed.tech/tags/workplace.md>)

### AI overview

HubEgo-100K v1.0 releases 10,124 hours of 1080p egocentric skilled-work video, recorded by 12,227 contributors across 15,548 environments, with synchronized 100 Hz IMU data from a head-mounted phone camera. Its 72,754 episodes span 11 setting types, 30 work categories, 56 task types and 35 action families; 83% of hours are workplace recordings. Episodes include machine-generated action and language annotations and are available in LeRobot and MCAP formats under CC BY 4.0. The authors measure diversity across these dimensions and report that new contributors and words continue to appear as randomly ordered episodes are added. They relate dataset properties, including hand visibility and inertial sensing, to published evidence on six properties tied to robot policy performance. This version documents collection, curation and privacy measures, but does not yet include the benchmarks or later releases planned for the project.

### Source excerpt

arXiv:2611.21938v1 Announce Type: cross Abstract: Robot policies trained by imitation generalize poorly outside the environments and objects seen in training. Published data-scaling studies find that generalization grows with the diversity of training environments and objects rather than with more demonstrations per environment. Robot data is costly to collect across many environments, whereas egocentric human video can be recorded in the wild at scale; existing egocentric video, however, mostly shows daily-life activity and contains little skilled work in real workplaces. We present HubEgo-100K, a dataset built for diversity. This version of the paper covers the first release, v1.0: 10,124 hours of 1080p egocentric video with synchronized 100 Hz inertial (IMU) data from one head-mounted phone camera. In v1.0, 12,227 contributors record their own unscripted work in 15,548 environments (each one contributor's own setting of one type, such as their kitchen or workshop) across 11 setting types, 30 work categories, 56 task types and 35 action families, with 83% of hours in workplace settings. The release comprises 72,754 episodes with machine-generated action and language annotations, in LeRobot and MCAP formats under CC BY 4.0. We measure diversity along each axis, show that inside v1.0, with episodes in random order, new contributors and words keep appearing, and relate the measurements to published evidence on six data properties tied to policy performance, including hand visibility and inertial sensing. We also document collection, curation, privacy measures and data format, and provide a datasheet. Later versions of this paper will add benchmarks and cover further releases toward 100,000 hours.

## Scaling Laws for Vision-Language Fusion

DevFeed: [Scaling Laws for Vision-Language Fusion](<https://devfeed.tech/articles/scaling-laws-for-vision-language-fusion-141769.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22351>)

Author: Seng Pei Liew, Teppei Suzuki, Seiya Tokui

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [vlm](<https://devfeed.tech/topics/vlm.md>), [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>), [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>)

Tags: [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [fusion](<https://devfeed.tech/tags/fusion.md>), [laws](<https://devfeed.tech/tags/laws.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [scaling-laws](<https://devfeed.tech/tags/scaling-laws.md>), [vision](<https://devfeed.tech/tags/vision.md>)

### AI overview

The paper proposes a scaling law for vision-language pre-training that separates modality-specific learning trajectories from shifts caused by competition between image and text data. It reports that the law predicts validation losses across data mixture ratios, training steps, and model sizes. Under a fixed data budget, early fusion preserves text performance while late fusion favors vision performance; the reported trade-off tilts increasingly toward early fusion as model size grows. The abstract says experiments support the law's ability to capture these dynamics and inform fusion choices.

### Source excerpt

arXiv:2611.22351v1 Announce Type: new Abstract: Vision-language models (VLMs) have emerged as central foundation models combining multimodal perception with text comprehension. While VLMs are traditionally built via late fusion by continued pre-training an off-the-shelf language model on multimodal (image-text) data, mixing modalities from the start of pre-training (early fusion) has recently gained traction. However, the fundamental trade-offs between these paradigms remain poorly understood, resulting in expensive and heuristic-driven training recipes. In this paper, we systematically analyze the learning dynamics of these multimodal pre-training processes, showing that validation losses during vision-language fusion can be modeled by separating the base learning trajectory, where different data types contribute at distinct rates, from a distribution shift driven by the competition between modalities. We derive a scaling law that unifiedly describes these dynamics, enabling accurate loss predictions across mixture ratios, training steps, and model sizes. Crucially, our law characterizes the trade-off between early and late fusion: under a fixed data budget, early fusion preserves text performance while late fusion favors vision performance, yielding a trade-off that increasingly tilts toward early fusion as model size scales. Extensive experiments confirm that our scaling law reliably captures multimodal learning dynamics and governs fusion decisions.

## AdaFlowLite: Scalable and Non-blocking Inference on Asynchronous Mobile Data

DevFeed: [AdaFlowLite: Scalable and Non-blocking Inference on Asynchronous Mobile Data](<https://devfeed.tech/articles/adaflowlite-scalable-and-non-blocking-inference-on-asynchronous-mobile-data-139897.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21691>)

Author: Sicong Liu, Fengmin Wu, Yuan Gao, Bin Guo, Zimu Zhou, Hongkai Wen, Zhiwen Yu

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>), [async](<https://devfeed.tech/topics/async.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [adapting](<https://devfeed.tech/tags/adapting.md>), [adoption](<https://devfeed.tech/tags/adoption.md>), [affinity](<https://devfeed.tech/tags/affinity.md>), [also](<https://devfeed.tech/tags/also.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [approach](<https://devfeed.tech/tags/approach.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [based](<https://devfeed.tech/tags/based.md>), [cabins](<https://devfeed.tech/tags/cabins.md>), [cameras](<https://devfeed.tech/tags/cameras.md>), [can](<https://devfeed.tech/tags/can.md>), [challenge](<https://devfeed.tech/tags/challenge.md>), [consistency](<https://devfeed.tech/tags/consistency.md>), [control](<https://devfeed.tech/tags/control.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [inference](<https://devfeed.tech/tags/inference.md>), [mobile](<https://devfeed.tech/tags/mobile.md>), [mobile-data](<https://devfeed.tech/tags/mobile-data.md>)

### AI overview

AdaFlowLite proposes opportunistic inference for asynchronous mobile sensor data, starting inference as partial inputs arrive to avoid waiting for delayed modalities. It computes cross-modality affinity with a hierarchical, normalized matrix designed to handle varied and changing inputs, then uses a lightweight multimodal Swin Transformer to impute missing data in real time across modalities and downstream tasks without retraining. The abstract reports up to 80.4% lower inference latency, up to 62.1% higher accuracy, and nearly 50% lower energy use than status quo approaches. It also says the method can improve LLM performance when preprocessing asynchronous data; the excerpt gives no further details on that result.

### Source excerpt

arXiv:2611.21691v1 Announce Type: cross Abstract: The rise of mobile devices equipped with numerous sensors, such as LiDAR and cameras, has driven the adoption of multi-modal deep intelligence for distributed sensing tasks, such as smart cabins and driving assistance. However, the arrival time of mobile sensory data vary due to modality size and network dynamics, which can lead to delays (if waiting for slow data) or accuracy decline (if inference proceeds without waiting). Moreover, the diversity and dynamic nature of mobile systems exacerbate this challenge. In response, we present a shift to \textit{opportunistic} inference for asynchronous distributed multi-modal data, enabling inference as soon as partial data arrives. While existing methods focus on optimizing modality consistency and complementarity, known as modal affinity, they lack a \textit{computational} approach to control this affinity in open-world mobile environments. % \sysname pioneers the formulation of structured cross-modality affinity in mobile contexts using a hierarchical analysis-based normalized matrix. This approach accommodates the diversity and dynamics of modalities, generalizing across different types and numbers of inputs. Employing an multi-modal lightweight Swin Transformer (MMLST), \sysname facilitates real-time and flexible data imputation, adapting to various modalities and downstream tasks without retraining. Experiments show that \sysname significantly reduces inference latency by up to 80.4\% and enhances accuracy by up to 62.1\%, while achieving nearly a 50\% reduction in energy consumption, outperforming status quo approaches. Also, this method can enhance LLM performance to preprocess asynchronous data.

## Liquid AI Releases d1.2 and d1.2-ultra: Omnimodal Decision Models That Top the AA Decision Index

DevFeed: [Liquid AI Releases d1.2 and d1.2-ultra: Omnimodal Decision Models That Top the AA Decision Index](<https://devfeed.tech/articles/liquid-ai-releases-d1-2-and-d1-2-ultra-omnimodal-decision-models-that-top-the-aa-decision-index-137110.md>)

Original publisher: [Read original article](<https://www.marktechpost.com/2026/11/17/liquid-ai-d1-2-omnimodal-decision-models/>)

Author: Asif Razzaq

Published: 2026-11-17T09:20:57Z

Content type: release

Language: en

Sources: [MarkTechPost](<https://devfeed.tech/sources/marktechpost.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>), [decision-making](<https://devfeed.tech/topics/decision-making.md>)

Tags: [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-infrastructure](<https://devfeed.tech/tags/ai-infrastructure.md>), [ai-shorts](<https://devfeed.tech/tags/ai-shorts.md>), [api](<https://devfeed.tech/tags/api.md>), [applications](<https://devfeed.tech/tags/applications.md>), [artificial-analysis](<https://devfeed.tech/tags/artificial-analysis.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [audio](<https://devfeed.tech/tags/audio.md>), [context-engineering](<https://devfeed.tech/tags/context-engineering.md>), [decision](<https://devfeed.tech/tags/decision.md>), [decision-models](<https://devfeed.tech/tags/decision-models.md>), [editors-pick](<https://devfeed.tech/tags/editors-pick.md>), [enterprise-ai](<https://devfeed.tech/tags/enterprise-ai.md>), [images](<https://devfeed.tech/tags/images.md>), [index](<https://devfeed.tech/tags/index.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [liquid-ai](<https://devfeed.tech/tags/liquid-ai.md>), [new-releases](<https://devfeed.tech/tags/new-releases.md>), [releases](<https://devfeed.tech/tags/releases.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>), [staff](<https://devfeed.tech/tags/staff.md>), [tech-news](<https://devfeed.tech/tags/tech-news.md>), [technology](<https://devfeed.tech/tags/technology.md>), [text](<https://devfeed.tech/tags/text.md>), [tokens](<https://devfeed.tech/tags/tokens.md>), [zero](<https://devfeed.tech/tags/zero.md>)

### AI overview

Liquid AI's d1.2 family accepts text, images and audio, returning probabilities over fixed answers without generating output tokens. The excerpt reports d1.2-ultra scored 75 on Artificial Analysis Decision Index v2.1, ahead of d1.2 at 69; Liquid AI reports calibration errors of 0.8 and 1.2 points, respectively, across 13 benchmarks, based on its own tests. Five added answer types handle value extraction, span finding, object boxes, ranged estimates and multiple applicable choices. On held-out audio recordings, both models answered 95-97% correctly; d1.2 outperformed ultra on multi-label picks, 90% to 87%. d1.2 is described as faster and less than half the price of ultra. Both are hosted through Liquid's API; weights, parameter counts and latency figures are unavailable in the excerpt.

### Source excerpt

Liquid AI released d1.2 and d1.2-ultra, decision models that read text, images and audio and return calibrated probabilities with zero output tokens. d1.2-ultra scores 75 on the Artificial Analysis Decision Index v2.1. The post Liquid AI Releases d1.2 and d1.2-ultra: Omnimodal Decision Models That Top the AA Decision Index appeared first on MarkTechPost.

## T2T Technical Report: Synthesizing GUI Agent Trajectories at Scale from Video Tutorials

DevFeed: [T2T Technical Report: Synthesizing GUI Agent Trajectories at Scale from Video Tutorials](<https://devfeed.tech/articles/t2t-technical-report-synthesizing-gui-agent-trajectories-at-scale-from-video-tutorials-137330.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18715>)

Author: Qingsong Xie, Yanlin Fei, Nazhou Liu, Jiacheng Tang, Madalina Ciobanu, Navan Preet Singh, Qingqing Mao, Ritankar Das

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [computer-use](<https://devfeed.tech/topics/computer-use.md>), [gui-agents](<https://devfeed.tech/topics/gui-agents.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>)

Tags: [actions](<https://devfeed.tech/tags/actions.md>), [agent](<https://devfeed.tech/tags/agent.md>), [annotation](<https://devfeed.tech/tags/annotation.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [automated](<https://devfeed.tech/tags/automated.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [capabilities](<https://devfeed.tech/tags/capabilities.md>), [covering](<https://devfeed.tech/tags/covering.md>), [cross](<https://devfeed.tech/tags/cross.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [desktop](<https://devfeed.tech/tags/desktop.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [environments](<https://devfeed.tech/tags/environments.md>), [evaluations](<https://devfeed.tech/tags/evaluations.md>), [execution](<https://devfeed.tech/tags/execution.md>), [fine](<https://devfeed.tech/tags/fine.md>), [generalist](<https://devfeed.tech/tags/generalist.md>), [gui](<https://devfeed.tech/tags/gui.md>), [scale](<https://devfeed.tech/tags/scale.md>), [tutorials](<https://devfeed.tech/tags/tutorials.md>), [video](<https://devfeed.tech/tags/video.md>)

### AI overview

T2T turns unannotated desktop, web, and mobile tutorial videos into structured, executable GUI-agent trajectories without human annotation. It filters video segments, identifies keyframes through localized UI changes, selects observation frames without shortcuts, grounds actions with awareness of whether UI elements are present, and normalizes actions in two stages. The resulting data supports training views for navigation, planning, forward dynamics, and grounding. Fine-tuning Qwen3-VL-8B on a subset of more than one million generated samples produced reported gains on OSWorld-G, ScreenSpot-Pro, AndroidControl, and AgentNetBench, despite using no in-distribution training data from the target domains. The abstract reports improved grounding, step-level execution, trajectory decisions, and cross-application and cross-task generalization; it gives no quantitative results or implementation details.

### Source excerpt

arXiv:2611.18715v1 Announce Type: cross Abstract: Training generalist GUI agents requires large volumes of interaction trajectories comprising screenshot sequences paired with grounded UI actions, yet collecting such data through human demonstration is expensive and difficult to scale. Although public software tutorial videos contain rich operational demonstrations, their continuous pixel streams lack discrete action boundaries, semantic intents, and precise spatial coordinates, making them difficult to use directly for model training. To address this, we present T2T (Tutorial-to-Trajectory), a fully automated pipeline for synthesizing GUI agent trajectories at scale. T2T spans desktop, web, and mobile platforms to convert unannotated video tutorials into structured, executable interaction trajectories. To guarantee trajectory quality, T2T combines active segment filtering, keyframe identification via localized UI change, shortcut-free observation frame selection, presence-aware grounding, and two-stage action-space normalization. These refined trajectories are then converted into multi-task training views spanning navigation, planning, forward dynamics, and grounding. We fine-tune Qwen3-VL-8B on a subset of the more than one million human-annotation-free training samples generated by this pipeline. Evaluations show that the model achieves consistent gains across multiple benchmarks covering desktop, web, and mobile environments, including OSWorld-G, ScreenSpot-Pro, AndroidControl, and AgentNetBench, indicating improved core capabilities in grounding, step-level instruction execution, and trajectory-level decision-making. Notably, all gains are achieved without any in-distribution training data from target domains. Learning exclusively from trajectories extracted from public videos via the T2T pipeline, the model exhibits cross-application and cross-task generalization capabilities.

## Sage S2.1 Flash Technical Report

DevFeed: [Sage S2.1 Flash Technical Report](<https://devfeed.tech/articles/sage-s2-1-flash-technical-report-137651.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21072>)

Author: Kun Cai, Haodong Chen, Guangran Cheng, Qingxiu Cheng, Xu Dai, Guanting Dong, Qiming Ge, Lixin Gu, Tao Gui, Honglin Guo, Yuyang Hu, Zixian Huang, Jiajie Jin, Alexander Lam, Jiaqiang Li, Yining Li, Jiah

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>), [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [agent](<https://devfeed.tech/tags/agent.md>), [artifacts](<https://devfeed.tech/tags/artifacts.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [bench](<https://devfeed.tech/tags/bench.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [browser](<https://devfeed.tech/tags/browser.md>), [capabilities](<https://devfeed.tech/tags/capabilities.md>), [coding](<https://devfeed.tech/tags/coding.md>), [comparison](<https://devfeed.tech/tags/comparison.md>), [completion](<https://devfeed.tech/tags/completion.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [deepresearch](<https://devfeed.tech/tags/deepresearch.md>), [desktop](<https://devfeed.tech/tags/desktop.md>), [develop](<https://devfeed.tech/tags/develop.md>), [documentary](<https://devfeed.tech/tags/documentary.md>), [domains](<https://devfeed.tech/tags/domains.md>), [flash](<https://devfeed.tech/tags/flash.md>), [report](<https://devfeed.tech/tags/report.md>), [sage](<https://devfeed.tech/tags/sage.md>), [technical](<https://devfeed.tech/tags/technical.md>)

### AI overview

Sage S1.5 Flash is a 320-billion-parameter mixture-of-experts model built to reason over scientific evidence and act in interactive environments. Its Sage Sci-Perception component uses dedicated encoders for molecular graphs and biological sequences, and represents spectra as ordered measurement patches; modality-specific projectors feed these inputs into a shared backbone alongside text and images. Agent training uses verifiable tool-use, search, and coding environments, plus workflows for machine-learning experiments and scientific software, with completion checked through execution results, environment changes, or submitted artifacts. Across 33 benchmarks in seven domains, it ranked first against five baselines on OfficeQA Pro, MLE-bench Lite, Harvey LAB, and SpreadsheetBench 2, and second on OSWorld-Verified, WebArena-Verified, and DeepResearch Bench II. The abstract reports strengths in documentary evidence, verifiable workbooks, and machine-learning tasks, with competitive desktop and browser performance.

### Source excerpt

arXiv:2611.21072v1 Announce Type: new Abstract: We introduce Sage S1.5 Flash, a 320-billion-parameter general-purpose mixture-of-experts model designed to reason over scientific evidence and act in interactive environments. Sage Sci-Perception uses dedicated encoders for molecular graphs and biological sequences and represents spectra as ordered patches of measurements. Modality-specific projectors map these representations into the input space of a shared backbone, which processes them alongside text and images. To develop the model's agent capabilities, we construct verifiable environments for tool use, search, and coding, as well as specialized workflows involving machine learning experiments and scientific software. Task completion is checked against execution results, changes in environment state, or submitted artifacts. We evaluate Sage S1.5 Flash on 33 benchmarks across seven domains. In the primary comparison against five baselines, it ranks first on OfficeQA Pro (72.2), MLE-bench Lite (90.3 HumanRank), Harvey LAB, and SpreadsheetBench~2, and second on OSWorld-Verified (85.2), WebArena-Verified (67.6), and DeepResearch Bench~II (54.2). These results show particular strengths in using documentary evidence, producing verifiable workbooks, and carrying out machine learning tasks, alongside competitive performance in desktop and browser workflows that require the model to act on visual feedback.

## D\\'ecoupage: Directorial Audio-Visual Control for Long Multi-Shot Video Generation

DevFeed: [D\\'ecoupage: Directorial Audio-Visual Control for Long Multi-Shot Video Generation](<https://devfeed.tech/articles/d-ecoupage-directorial-audio-visual-control-for-long-multi-shot-video-generation-137353.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18866>)

Author: Yifan Wang, Hao Ouyang, Qiuyu Wang, Yihao Meng, Wen Wang, Zeyue Tian, Yichong Lu, Hanlin Wang, Qingyan Bai, Zichen Liu, Yujun Shen, Qifeng Chen

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Generative AI](<https://devfeed.tech/topics/generative-ai.md>), [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [audio](<https://devfeed.tech/tags/audio.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [control](<https://devfeed.tech/tags/control.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-sd](<https://devfeed.tech/tags/cs-sd.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [framework](<https://devfeed.tech/tags/framework.md>), [generation](<https://devfeed.tech/tags/generation.md>), [generative](<https://devfeed.tech/tags/generative.md>), [independent](<https://devfeed.tech/tags/independent.md>), [mechanism](<https://devfeed.tech/tags/mechanism.md>), [multi-shot-video-generation](<https://devfeed.tech/tags/multi-shot-video-generation.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [schedule](<https://devfeed.tech/tags/schedule.md>), [temporal](<https://devfeed.tech/tags/temporal.md>), [timing](<https://devfeed.tech/tags/timing.md>), [video](<https://devfeed.tech/tags/video.md>), [video-generation](<https://devfeed.tech/tags/video-generation.md>), [visibility](<https://devfeed.tech/tags/visibility.md>)

### AI overview

DECOUPAGE is a screenplay-driven framework for generating 15- to 60-second, multi-shot audio-video scenes with independently timed visual and acoustic tracks. It uses persistent character identities, dual-schedule temporal cross-attention, and a directorial lip-sync mechanism that separates speech conditioning from facial motion during off-screen dialogue. The authors report that, on multi-shot conversational benchmarks, it controls shot transitions, audio timing, and visibility-aware lip synchronization, addressing unintended speech during silence and mouth movement on silent listeners. The evidence is limited to the supplied abstract; it does not provide benchmark details or comparative results.

### Source excerpt

arXiv:2611.18866v1 Announce Type: new Abstract: In cinematic storytelling, sound and visual transitions do not always coincide in lockstep: dialogue frequently bridges across cuts to reveal reactions or introduces new scenes via J- and L-cuts, while deliberate silences pace the scene. Existing audio-visual generative frameworks, however, lack explicit mechanisms to coordinate independent visual and acoustic timing over extended horizons. As narrative duration and shot complexity scale, these systems struggle to maintain long-range dialogue pacing, often hallucinating speech during intended silences or erroneously driving mouth movements on silent listeners. To enable fine-grained cinematic control, we present DECOUPAGE, a screenplay-driven framework for long-form, multi-shot, and multi-character joint audio-video generation. Rather than coupling modality boundaries by default, DECOUPAGE uncouples generation into independently bounded visual and acoustic tracks governed by persistent character identities. DECOUPAGE builds on a joint audio-video diffusion model with dual-schedule temporal cross-attention, incorporating a directorial Lip-sync mechanism that selectively decouples speech conditioning from visual facial dynamics during off-screen dialogue. On multi-shot conversational benchmarks, DECOUPAGE achieves precise control over shot transitions, audio timing, and visibility-aware lip synchronization, directing 15- to 60-second screenplay-driven audio-video generation with multimodal coherence.

## VIVID: A Visual Harness for Agentic Video Editing

DevFeed: [VIVID: A Visual Harness for Agentic Video Editing](<https://devfeed.tech/articles/vivid-a-visual-harness-for-agentic-video-editing-137602.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20688>)

Author: Wenhan Yu, Zhaoxi Zhang, Zhilong Tan, Weikang Li, Lei Sha

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Agent Harness](<https://devfeed.tech/topics/agent-harness.md>), [agentic-coding](<https://devfeed.tech/topics/agentic-coding.md>), [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>), [Claude Code](<https://devfeed.tech/topics/claude-code.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [agents](<https://devfeed.tech/tags/agents.md>), [assembly](<https://devfeed.tech/tags/assembly.md>), [audio](<https://devfeed.tech/tags/audio.md>), [audio-analysis](<https://devfeed.tech/tags/audio-analysis.md>), [code](<https://devfeed.tech/tags/code.md>), [code-execution](<https://devfeed.tech/tags/code-execution.md>), [codex](<https://devfeed.tech/tags/codex.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-mm](<https://devfeed.tech/tags/cs-mm.md>), [harness](<https://devfeed.tech/tags/harness.md>), [improvement](<https://devfeed.tech/tags/improvement.md>), [localization](<https://devfeed.tech/tags/localization.md>), [model-training](<https://devfeed.tech/tags/model-training.md>), [opencode](<https://devfeed.tech/tags/opencode.md>), [post-production](<https://devfeed.tech/tags/post-production.md>), [repair](<https://devfeed.tech/tags/repair.md>), [tools](<https://devfeed.tech/tags/tools.md>), [transcription](<https://devfeed.tech/tags/transcription.md>), [video](<https://devfeed.tech/tags/video.md>), [video-editing](<https://devfeed.tech/tags/video-editing.md>)

### AI overview

VIVID adds media tools and evidence tracking to native coding agents for video editing, without additional model training. Agents can inspect video, analyze and transcribe audio, retain observations with media versions and timestamps, make edits with their coding and shell tools, and review rendered outputs for revisions. On 100 AgenticVBench tasks across Assembly, Repair, Sequencing, and Repurpose, Codex with VIVID scored 58.6%, 20.2 percentage points above native Codex. The harness also scored 55.1% with Claude Code and 37.1% with OpenCode, each substantially above its native baseline.

### Source excerpt

arXiv:2611.20688v1 Announce Type: new Abstract: Video editing requires agents to make editing decisions grounded in audiovisual observations and precise temporal localization, and to evaluate their effects in rendered outputs. General-purpose coding agents possess strong comprehension, reasoning, and code execution capabilities, while media interaction and evidence management mechanisms tailored to video tasks can further unlock their editing potential. We introduce VIVID, a visual harness that supports multiple native coding agents in video post-production without additional model training. VIVID provides tools for on-demand video inspection, audio analysis, and transcription, and preserves observations alongside their corresponding media versions and temporal coordinates. Agents can actively retrieve these observations as they reason, perform edits using their native coding and shell tools, and inspect rendered outputs to guide subsequent revisions. On AgenticVBench, comprising 100 tasks across Assembly, Repair, Sequencing, and Repurpose, Codex with VIVID achieves an average score of 58.6%, an improvement of 20.2 percentage points over native Codex. The same harness also integrates with Claude Code and OpenCode, achieving average scores of 55.1% and 37.1%, respectively, both substantially exceeding their native baselines. These results highlight VIVID's potential as a reusable visual harness for extending the capabilities of general-purpose coding agents to video editing.

[Next page](<https://devfeed.tech/topics/multimodal-ai.md?cursor=WyIyMDI2LTExLTE3VDA1OjAwOjAwKzAwOjAwIiwgImFiOGQ4NzU0LTlkM2QtNDM3MS1iZDkyLTg5ZGRhNTVhZDEzNCJd>)