# multimodal Published articles for multimodal. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## NASA-IBM Lunar Foundation Model Goes Open Source With a 2M-Tile Dataset and 22% Lower Ice-Mapping Error DevFeed: [NASA-IBM Lunar Foundation Model Goes Open Source With a 2M-Tile Dataset and 22% Lower Ice-Mapping Error]() Original publisher: [Read original article]() Author: Harold Fritts Published: 2026-09-14T16:43:16Z Content type: news Language: en Sources: [StorageReview.com]() Topics: [lunar foundation model](), [foundation-models](), [ibm](), [Open Source](), [hugging face](), [dataset](), [multimodal](), [Benchmark]() Tags: [ai](), [benchmark](), [data](), [enterprise](), [fine-tuning](), [foundation-models](), [hugging-face](), [ibm](), [lunar-foundation-model](), [machine-learning](), [multimodal](), [nasa](), [open-source](), [space]() ### AI overview IBM and NASA have released the NASA-IBM Lunar Foundation Model as open source on Hugging Face, along with its weights, technical report, and training dataset. Built on TerraMind, the model uses multimodal lunar observations for tasks including ice-deposit mapping, volcanic-feature detection, and crater detection. Reported benchmarks show up to 22% lower ice-mapping error than SwinV2-B, while the accompanying dataset contains roughly 2 million image tiles from nine instruments across four lunar missions. ### Source excerpt IBM and NASA have released the NASA-IBM Lunar Foundation Model as open source, one of the first publicly available foundation models built for scientific study of the Moon. The weights, a technical report, and the machine-learning-ready dataset it was trained on are up on Hugging Face under the Prithvi family, which already covers Earth observation, The post NASA-IBM Lunar Foundation Model Goes Open Source With a 2M-Tile Dataset and 22% Lower Ice-Mapping Error appeared first on StorageReview.com. ## On-Device AI Series (Part 5): LiteRT-LM DevFeed: [On-Device AI Series (Part 5): LiteRT-LM]() Original publisher: [Read original article]() Author: Oğuzhan Aslan Published: 2026-09-14T05:59:12Z Content type: tutorial Language: en Sources: [ProAndroidDev - Medium]() Topics: [LiteRT](), [On-device AI](), [Large Language Model](), [Inference](), [Streaming](), [GPU](), [LLM evaluation / benchmarking](), [PyTorch]() Tags: [ai](), [android](), [android-development](), [benchmarking](), [gpu](), [inference](), [litert](), [llama](), [llm](), [multimodal](), [on-device](), [on-device-ai](), [programming](), [pytorch](), [qwen](), [software-development](), [streaming]() ### AI overview This tutorial explains LiteRT-LM for running large language models on-device. It covers the Engine/Session API, streaming output, system prompts, tool calling, multimodal inputs, thinking mode, and CPU-versus-GPU benchmarking. The article also discusses tradeoffs involving privacy, network independence, latency, memory, sampling configuration, and model capability compared with cloud APIs. ### Source excerpt Put your phone in airplane mode. Open the app, type a question, and watch the answer arrive one token at a time -- no spinner waiting on a network round-trip, no API key, no per-token bill, and nothing you typed ever leaving the device. LiteRT-LM removes the genuinely hard parts of running an LLM on-device -- KV-cache management, token streaming, backend selection -- but it doesn't remove your job so much as relocate it. What's left on your plate is a short, specific list: sizing a combined input+output token budget, owning your own sampling defaults, hand-building system prompts and tool calling out of raw text, and one native-library collision that presents as a SIGSEGV rather than a build error. Know those going in and the API itself is a clean three-step pattern. We'll get there in that order: Why you'd choose this runtime and what it costs you versus the cloud. The Engine/Session model you need to read the code at all. Real implementation samples -- streaming, system prompts and tool calling, multimodal inputs, thinking mode, and CPU-vs-GPU benchmarking. The anti-patterns to avoid. A developer-friendliness rating on the same rubric as Parts 1-4. Why Use LiteRT-LM? You reach for LiteRT-LM instead of hand-rolling generation on top of raw LiteRT when: You need multi-turn conversation, not single-shot inference -- session state and KV-cache bookkeeping are handled for you, and resetting a conversation is a session swap, not a model reload. You need streaming output -- token-by-token delivery for a responsive chat UI, instead of a blocking call that returns everything at once. You're choosing between CPU and GPU per device -- the explicit backend parameter turns that into a runtime decision instead of a build-time guess. You want a pre-converted model without doing your own PyTorch-to-LiteRT conversion work -- the Model Zoo covers Gemma, Qwen, Llama, and more out of the box. You're willing to own sampling -- the engine won't pick sane decoding defaults for you; that's on the ## Enterprise-Grade Precision for Long-Context Multimodal Embedding Inference on Cloud TPU DevFeed: [Enterprise-Grade Precision for Long-Context Multimodal Embedding Inference on Cloud TPU]() Original publisher: [Read original article]() Author: Anthony Su; Injae Kwak Published: 2026-09-12T11:04:33.891311Z Content type: article Language: en Sources: [Google Developers Blog]() Topics: [Embeddings](), [vllm](), [Inference](), [long-context](), [autoscaling](), [Kubernetes](), [Cloud](), [Google](), [Artificial Intelligence](), [model-deployment](), [Open Source]() Tags: [autoscaling](), [embedding](), [github](), [gpu](), [inference](), [kubernetes](), [long-context](), [multimodal](), [open-source](), [retrieval](), [tpu](), [vllm]() ### AI overview This article explains how Google Cloud integrated native TPU support into vLLM to serve long-context, multimodal embedding models at production scale. It describes elastic TPU provisioning with Google Kubernetes Engine, autoscaling across accelerator types, and TPU-specific optimizations for long sequences and chunked prefill. The resulting setup is designed to preserve numerical parity with GPU reference baselines while supporting high-throughput semantic retrieval applications. ### Source excerpt Google Cloud has natively integrated TPU support into the vLLM serving engine, allowing developers to elastically scale high-demand embedding pipelines using Google Kubernetes Engine (GKE). To handle massive 15K+ token contexts for models like Qwen3-Embedding-8B, the engineering team implemented TPU-specific optimizations such as hardware-safe tensor alignment, JAX/XLA compilation pre-warming, and a hybrid StepPool architecture for chunked prefill management. These enhancements achieve near-perfect numerical parity with reference GPU baselines, and developers can immediately leverage the open-sourced setup recipes on the AI-Hypercomputer GitHub to build their own high-throughput semantic retrieval applications. ## Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering DevFeed: [Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering]() Original publisher: [Read original article]() Published: 2026-09-11T00:00:00Z Content type: article Language: en Sources: [Apple Machine Learning Research]() Topics: [LLM evaluation / benchmarking](), [Ground truth / benchmark quality](), [Hallucination detection]() Tags: [benchmark](), [computer-vision](), [evaluation](), [metrics](), [multimodal](), [research](), [video]() ### AI overview The article introduces CapQuiz, a reference-free benchmark for evaluating video-caption quality through human-verified multiple-choice questions. It also proposes CapF1, combining factuality and visual-information coverage, and reports stronger correlation with human judgments than existing metrics. ### Source excerpt Evaluating video captioning remains a critical challenge for Visual Large Language Models (VLLMs). Existing metrics primarily rely on matching generated text against ground-truth references. This paradigm suffers from the "one-to-many" nature of video description, where high-quality captions are often penalized for lexical mismatches or valid shifts in visual focus. Furthermore, such assessments are typically one-dimensional, failing to provide a fine-grained analysis of caption quality. To address this, we redefine caption quality via information fidelity: A caption must maximize the coverage... ## SimpleDesign: A Joint Model for Protein Sequence and Structure Codesign DevFeed: [SimpleDesign: A Joint Model for Protein Sequence and Structure Codesign]() Original publisher: [Read original article]() Published: 2026-09-11T00:00:00Z Content type: article Language: en Sources: [Apple Machine Learning Research]() Topics: [multimodal](), [Transformer](), [datasets]() Tags: [benchmarks](), [drug-discovery](), [generation](), [model](), [multimodal](), [training]() ### AI overview SimpleDesign is a single-stage, end-to-end multimodal generative model for jointly designing protein sequences and three-dimensional structures. It uses Transformer-based multimodal backbones, trains directly in data space on more than 2 million sequence-structure pairs, and achieves competitive results on co-design and unconditional generation benchmarks. ### Source excerpt Proteins are fundamental to biological processes, with their function determined by the complex interplay between the amino acid sequence and the three-dimensional structure. Developing generative models capable of understanding this intrinsically multi-modal relationship is crucial for fields like drug discovery and protein engineering. Existing models often rely on a multi-stage training process where autoencoders that tokenize data into latent representations are trained in a first stage. Secondly, a generative model is trained on the latent representation of the autoencoder(s), i.e... ## Building Pinterest's VLM Serving Stack on NVIDIA Dynamo DevFeed: [Building Pinterest's VLM Serving Stack on NVIDIA Dynamo]() Original publisher: [Read original article]() Author: Pinterest Engineering Published: 2026-09-10T23:08:16Z Content type: article Language: en Sources: [Pinterest Engineering Blog - Medium]() Topics: [vlm](), [Inference](), [model-deployment](), [Orchestration](), [Latency](), [Caching](), [Open Source Models & Datasets]() Tags: [blackwell](), [cache](), [dynamo](), [embeddings](), [engineering](), [inference](), [latency](), [llm](), [multimodal](), [multimodal-ai](), [nvidia](), [nvidia-blackwell](), [open-source](), [orchestration](), [performance](), [pinterest](), [vllm](), [vlm](), [vlm-serving]() ### AI overview Pinterest describes its VLM serving stack built on NVIDIA Blackwell GPUs and NVIDIA Dynamo. The stack addresses multimodal inference demands such as image processing, variable prefill costs, KV-cache pressure, routing, and cache offloading. ### Source excerpt Lei Pan | Senior Software Engineer; Salina Wu | Senior Software Engineer; Cristian Lopez | Software Engineer I; Guangtong Bai | Staff Software Engineer; Soam Acharya | Principal Engineer; Saurabh Vishwas Joshi | Principal Engineer; Chia-Wei Chen | Staff Software Engineer; Ambud Sharma | Principal Engineer Why VLM Serving Matters at Pinterest Pinterest is a visual search and discovery platform, so its AI systems must reason over both language and visual content. Vision-language models (VLMs), which can interpret images, compare visual candidates, and respond naturally to user intent, are becoming the foundation for the next generation of Pinterest experiences: Pinterest Assistant, hybrid search, multimodal reranking, content understanding, signal generation, content safety, and more. This direction also reflects Pinterest's broader strategy to customize open-source models to meet its product & scale needs. Pinterest Assistant is a standout example. This multi-turn conversational experience covers both user language and visual content. Serving it requires low-latency VLM inference over rich multimodal context as well as reworking Qwen3-VL with proprietary multimodal embeddings to cut runtime cost while improving performance. Serving VLMs, however, introduces more challenges compared to text-only LLM workloads. Requests may carry multiple images, require extra vision encoder computation, incur larger and more variable prefill cost, and create higher KV cache pressure. To support this new class of models & product experiences, we built Pinterest's VLM serving stack on top of NVIDIA Blackwell GPUs and NVIDIA Dynamo. Blackwell GPUs incorporate many architectural innovations that are uniquely positioned for today's most demanding AI workloads -- including higher BF16/FP8 compute throughput, increased memory bandwidth, and larger HBM memory capacity -- that enable dramatically higher performance for inference. Dynamo provides a distributed inference orchestration layer that g ## Video and image search in Amazon Bedrock Knowledge Base using Marengo 3.0 DevFeed: [Video and image search in Amazon Bedrock Knowledge Base using Marengo 3.0]() Original publisher: [Read original article]() Author: Eric Kim Published: 2026-09-10T21:15:39Z Content type: tutorial Language: en Sources: [Artificial Intelligence]() Topics: [AI search](), [Retrieval Augmented Generation (RAG)](), [Amazon S3](), [AWS IAM]() Tags: [amazon-bedrock](), [amazon-bedrock-knowledge-bases](), [announcements](), [audio](), [embedding](), [how-to](), [images](), [intermediate-200](), [multimodal](), [rag](), [s3](), [search](), [video]() ### AI overview A walkthrough for building an Amazon Bedrock Knowledge Base with TwelveLabs Marengo Embed 3.0 to perform natural-language semantic search across video, images, and audio. ### Source excerpt TwelveLabs Marengo Embed 3.0 is now generally available as an embedding model in Amazon Bedrock Knowledge Bases, bringing fully managed natural language search to video, image, and audio content. This walkthrough shows how to build a knowledge base powered by Marengo 3.0 and run semantic queries against your media. ## When to Use Encode-Prefill-Decode Disaggregation to Accelerate Multimodal Model Serving DevFeed: [When to Use Encode-Prefill-Decode Disaggregation to Accelerate Multimodal Model Serving]() Original publisher: [Read original article]() Author: Tanya Lenz Published: 2026-09-09T20:31:04Z Content type: article Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [Optimization](), [model-deployment](), [multimodal](), [Dynamo](), [Inference Performance](), [Embeddings](), [Mixture of Experts (MoE)](), [GPU](), [Open Source](), [Artificial Intelligence]() Tags: [agentic-ai-generative-ai](), [ai-agent](), [ai-inference](), [computer-vision-video-analytics](), [developer-tools-techniques](), [dynamo](), [embeddings](), [gpu](), [inference](), [low-latency-inference](), [mixture-of-experts-moe](), [model-serving](), [multimodal](), [nvfp4](), [open-source](), [optimization]() ### AI overview This article explains when encode-prefill-decode disaggregation improves multimodal model serving. Using NVIDIA Dynamo, it separates vision encoding from LLM prefill and decode so the stages can batch, schedule, and scale independently. The approach is most effective for image-heavy prompts, short-to-medium outputs, and quantized mixture-of-experts models, with reported gains of up to 5x in time to first token and 7x in end-to-end response time. ### Source excerpt Encode-prefill-decode (EPD) disaggregation is an inference optimization technique for multimodal models that separates the vision encoder stage from the prefill... ## NeoMME: an efficient Multimodal-native and Multilingual Encoder DevFeed: [NeoMME: an efficient Multimodal-native and Multilingual Encoder]() Original publisher: [Read original article]() Author: Tony Wu; Aurélien Lac Published: 2026-09-03T13:13:48Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [vlm](), [Fine-tuning](), [Large Language Model](), [GPU]() Tags: [apache](), [architecture](), [diffusion](), [embeddings](), [fine-tuning](), [gpu](), [hugging-face](), [multimodal](), [quantization](), [retrieval](), [training](), [transformers](), [vector](), [vision](), [vlm]() ### AI overview NeoMME is a family of multilingual multimodal encoders trained from scratch with a masked discrete-diffusion objective. It uses one bidirectional Transformer for text tokens and image patches, and is fine-tuned for visual document retrieval with dense and late-interaction embeddings. ### Source excerpt We introduce NeoMME, a family of 260M and 800M multilingual multimodal encoders. Unlike many generative visual language models, NeoMME does not use a separate pretrained vision tower or a causal language model. A single bidirectional Transformer processes both text tokens and raw image patches, and we train the entire model from scratch with a masked discrete-diffusion objective. We fine-tuned NeoMME for visual document retrieval using ColPali's page-image approach. ## From voice command to robotic arm: how agentic AI on the edge is changing the factory floor DevFeed: [From voice command to robotic arm: how agentic AI on the edge is changing the factory floor]() Original publisher: [Read original article]() Author: Arduino Team Published: 2026-09-01T12:20:24Z Content type: article Language: en Sources: [Arduino Blog]() Topics: [Artificial Intelligence](), [foundation-models](), [Automation](), [Arduino](), [UNO Q](), [AI Agent](), [Physical AI](), [multimodal]() Tags: [agentic-ai](), [ai](), [arduino](), [automation](), [featured](), [foundation-models](), [industrial](), [inference](), [latency](), [manufacturing](), [multimodal](), [robotic-arm](), [robotics](), [smart-factory](), [uno-q](), [usb](), [voice-commands](), [voice-control]() ### AI overview The article describes a demonstration in which Forgis uses a foundation model running on an Arduino UNO Q board to convert voice commands into robotic-arm actions. The system processes multimodal factory data and performs inference locally, enabling real-time control without a cloud round trip. ### Source excerpt For years, bringing real intelligence to industrial automation meant expensive infrastructure, proprietary systems, and steep learning curves. That's changing - fast. Foundation models powerful enough to run at the edge are turning natural language into machine control, and the factory floor is starting to look a lot more like a conversation. AI as the new [...] The post From voice command to robotic arm: how agentic AI on the edge is changing the factory floor appeared first on Arduino Blog. ## MAPS: Netflix's Multimodal Asset Personalization at Scale DevFeed: [MAPS: Netflix's Multimodal Asset Personalization at Scale]() Original publisher: [Read original article]() Author: Netflix Technology Blog Published: 2026-08-28T16:01:02Z Content type: article Language: en Sources: [Netflix](), [Netflix TechBlog - Medium]() Topics: [Netflix](), [personalization](), [Embeddings](), [multimodal](), [recommendation systems](), [data](), [A/B Testing]() Tags: [data](), [embedding](), [embeddings](), [machine-learning](), [model](), [multimodal](), [netflix](), [personalization](), [production](), [testing]() ### AI overview Netflix describes MAPS, a multimodal asset-personalization system that uses embeddings from artwork and video previews to address cold-start problems and personalize recommendations soon after a title launches. ### Source excerpt By Emma Yanyang Kong, Aditya Deshpande, Asad Abbasi, Bowei Yan, David Fagnan, Ashish Rastogi, Dhaval Patel, Ray Zhang Introduction The Netflix experience is a journey of discovery. Every visual cue, from the artwork on a title to the video previews that autoplay while you browse, is there to connect you with a story you will love. We call these visual cues assets, and choosing the right one for each member is a personalization problem of its own. But which image or video preview of Squid Game should we show you? And what do we do right after a title launches, when there's far too little interaction data to know which asset we should recommend to each member? For years, our models answered the first question well and the second poorly. They learned which assets members interacted with, but treated every asset as an opaque ID, blind to what was actually in the artwork or video preview. Right after a title launched, its assets had no history, so we dialed up exploration on its assets to gather interaction data, and otherwise fell back to popularity heuristics that ignore your taste. Only once enough interactions had piled up could personalization take over. This is the classic cold-start problem. This post shares how multimodal embeddings let our models see and hear the assets they recommend, so personalization can kick in far sooner, close to a title's launch. Because a new asset arrives with its embedding the model already understands, that embedding carries member taste signals from related assets immediately. Consequently, the model needs far less interaction history before it can personalize. We cover three production systems, artwork personalization, query-aware artwork ranking, and video preview personalization, plus a cheap trick for choosing new embeddings before committing to full end-to-end integration and A/B testing. Artwork Personalization A single image is often a member's first touchpoint with a title, so we create a diverse set of artworks for each tit ## Experiment with Qwen3.8-Flash-Next on NVIDIA GB300 NVL72 for Agentic Coding DevFeed: [Experiment with Qwen3.8-Flash-Next on NVIDIA GB300 NVL72 for Agentic Coding]() Original publisher: [Read original article]() Author: Michelle Horton Published: 2026-08-26T17:07:12Z Content type: article Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [qwen](), [long-context](), [Mixture of Experts (MoE)](), [agentic-coding](), [Inference](), [GPU](), [multimodal](), [NeMo](), [sglang](), [TensorRT-LLM](), [vllm](), [rust-ai]() Tags: [agentic-ai-generative-ai](), [agentic-coding](), [benchmarks](), [context-window](), [developer-tools-techniques](), [gb300-nvl72](), [inference](), [mixture-of-experts-moe](), [multimodal](), [nemo](), [nvlink](), [qwen](), [sglang](), [tensorrt-llm](), [top-stories](), [vllm]() ### AI overview This NVIDIA developer article introduces Qwen3.8-Flash-Next, a multimodal mixture-of-experts model released by Alibaba for experimentation and evaluation. It explains the model's long-context hybrid architecture, including Gated DeltaNet and Qwen Sparse Attention, and discusses reported efficiency improvements for million-token workloads. The article also covers inference support through SGLang, vLLM, TensorRT-LLM, and NVIDIA NeMo, plus performance on the NVIDIA GB300 NVL72 platform. ### Source excerpt Alibaba released the model weights for Qwen3.8-Flash-Next as a preview of the upcoming Qwen4 architecture for developers to experiment with and evaluate. It's... ## Luce: Relightable Gaussians for 3D Asset Generation DevFeed: [Luce: Relightable Gaussians for 3D Asset Generation]() Original publisher: [Read original article]() Published: 2026-08-26T00:00:00Z Content type: article Language: en Sources: [Apple Machine Learning Research]() Topics: [Artificial Intelligence](), [Benchmark]() Tags: [3d](), [ai](), [benchmark](), [computer-vision](), [generation](), [images](), [mesh](), [models](), [multimodal](), [techniques](), [vision]() ### AI overview Luce is a multimodal 3D representation for generating relightable assets from a single image. It combines geometry with physically based materials in a voxelized Gaussian cloud, compresses them into a material-aware latent space, and generates relightable PBR Gaussians and optional textured meshes. On Toys4K, it reports a 28% FID improvement over the strongest baseline and improves alignment on an AI-generated image benchmark. ### Source excerpt High-fidelity image-to-3D generation requires a 3D representation that captures both geometry and appearance. To support relighting and integration into standard rendering pipelines, the representation should include physically based rendering (PBR) modalities such as albedo, metallic-roughness, and surface normals. We propose Luce, a 3D representation that unifies geometry and PBR materials within a voxelized multimodal Gaussian cloud, using dedicated Gaussian primitives for each modality. A variational autoencoder compresses this representation into a unified material-aware latent space. A... ## GLM 5.3 Flash now available on AI Gateway DevFeed: [GLM 5.3 Flash now available on AI Gateway]() Original publisher: [Read original article]() Author: Jerilyn Zheng Published: 2026-08-26T00:00:00Z Content type: news Language: en Sources: [Vercel News]() Topics: [AI Chat]() Tags: [agent](), [ai](), [ai-gateway](), [coding](), [flash](), [images](), [language-models](), [model](), [multimodal](), [playground](), [streaming](), [vercel]() ### AI overview GLM 5.3 Flash from Z.ai is now available through AI Gateway. The multimodal coding model supports text and image inputs, function calling, structured output, streaming, and a 1M-token context window. ### Source excerpt GLM 5.3 Flash from Z.ai is now available on AI Gateway as zai/glm-5.3-flash. GLM 5.3 Flash is a multimodal coding model with a 1M-token context window. It accepts both text and image inputs and supports function calling, structured output, and streaming. To include images, pass them with text in a message. A request can include multiple images using URLs, Base64 data URLs, or binary data: To use GLM 5.3 Flash with a coding agent, run vercel ai-gateway coding-agents setup, then select zai/glm-5.3-flash as your agent's model. Try GLM 5.3 Flash in the model playground, or browse all language models. Read more ## AgentHands: Generating interactive hand gestures for spatially grounded agent conversations in XR DevFeed: [AgentHands: Generating interactive hand gestures for spatially grounded agent conversations in XR]() Original publisher: [Read original article]() Published: 2026-08-25T19:10:59Z Content type: article Language: en Sources: [The latest research from Google]() Topics: [Human-Computer Interaction and Visualization](), [AI Bots]() Tags: [3d](), [ai-agents](), [android-xr](), [human-computer-interaction-and-visualization](), [llm](), [machine-intelligence](), [multimodal](), [research-prototype]() ### AI overview AgentHands is an LLM-powered XR research prototype that adds synchronized hand gestures to conversational agents for spatially grounded guidance in physical tasks. ### Source excerpt Human-Computer Interaction and Visualization ## Building Federated Multimodal AI Workflows with NVIDIA FLARE DevFeed: [Building Federated Multimodal AI Workflows with NVIDIA FLARE]() Original publisher: [Read original article]() Author: Tanya Lenz Published: 2026-08-19T17:50:47Z Content type: article Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [vlm](), [SDKs]() Tags: [agentic-ai-generative-ai](), [ai](), [data-science](), [edge-computing](), [featured](), [federated-learning](), [multimodal](), [multimodal-ai](), [nvidia](), [python](), [streaming](), [training](), [training-ai-models](), [updates](), [vlms](), [workflows]() ### AI overview The article explains how NVIDIA FLARE supports federated training for multimodal and vision-language models when data remains distributed across sites. It focuses on deciding which model state to exchange and on efficiently transferring and aggregating large updates through externalization, tensor streaming, and disk-backed aggregation. ### Source excerpt Modern vision-language models (VLMs) can support tasks such as visual question answering, captioning, and image-text reasoning. In practice, however, the data... ## Multi-Vector (Late Interaction) Embedding Models with Sentence Transformers DevFeed: [Multi-Vector (Late Interaction) Embedding Models with Sentence Transformers]() Original publisher: [Read original article]() Author: Tom Aarsen; Antoine Chaffin; Raphael Sourty Published: 2026-08-18T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [sentence-transformers](), [Embeddings](), [Open Source]() Tags: [community](), [embedding](), [embeddings](), [guide](), [multimodal](), [nlp](), [open-source](), [retrieval](), [search](), [sentence-transformers](), [transformers](), [vector]() ### AI overview This article explains multi-vector, or late-interaction, embedding models with Sentence Transformers. It covers token-level representations, MaxSim scoring, retrieval over text and page images, integration with search systems, and index-size tradeoffs. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Seeing beyond BMI: Estimating cardiometabolic risk with smartphone imagery DevFeed: [Seeing beyond BMI: Estimating cardiometabolic risk with smartphone imagery]() Original publisher: [Read original article]() Published: 2026-08-17T10:34:00Z Content type: article Language: en Sources: [The latest research from Google]() Topics: [Deep neural networks](), [Deep learning](), [Machine learning](), [Google]() Tags: [deep-learning](), [general-science](), [google](), [health](), [health-bioscience](), [machine-intelligence](), [machine-learning](), [monitoring](), [multimodal](), [science]() ### AI overview Google Research presents PhotoScan, a deep learning approach that estimates body composition from smartphone photos and predicts insulin resistance with accuracy comparable to DXA scans in a clinical research setting. The article explains how body-composition measures such as fat distribution and visceral fat may complement wearable data for earlier cardiometabolic risk assessment. ### Source excerpt General Science ## Choosing & integrating LLM APIs: a practical guide DevFeed: [Choosing & integrating LLM APIs: a practical guide]() Original publisher: [Read original article]() Author: Cedric Turner Published: 2026-08-17T00:00:00Z Content type: tutorial Language: en Sources: [Redis Blog]() Topics: [Large Language Model](), [API](), [SDKs](), [Latency](), [Caching](), [Inference](), [Streaming](), [Embeddings](), [multimodal]() Tags: [apis](), [caching](), [code](), [engineering](), [inference](), [latency](), [llm](), [multimodal](), [sdks](), [streaming](), [tech-de]() ### AI overview A practical guide to choosing and integrating LLM APIs, covering provider capabilities, pricing, latency, quality evaluation, and the caching, retry, and memory layers needed for reliable production applications. ### Source excerpt Making your first LLM API call is easy: with most provider SDKs, it's about five lines of code. Keeping that call fast, affordable, and reliable once real users show up is where the actual engineering happens: costs can compound as conversations grow,... ## Elastic community newsletter -- August 2026 DevFeed: [Elastic community newsletter -- August 2026]() Original publisher: [Read original article]() Author: Elastic DevRel team Published: 2026-08-13T00:00:00Z Content type: news Language: en Sources: [Elastic Blog - Elasticsearch, Kibana, and ELK Stack]() Topics: [elasticsearch](), [devrel](), [dashboards](), [Embeddings](), [multimodal](), [quantization](), [AI search](), [tracing]() Tags: [audio](), [dashboards](), [devrel](), [elasticsearch](), [embeddings](), [images](), [multimodal](), [pdf](), [quantization](), [tracing](), [video]() ### AI overview The Elastic community newsletter covers Elasticsearch 9.5, including native PromQL, the Dashboards API, Columnar Mode, vector search improvements, multimodal semantic search, and Elastic Agent Builder tracing in technical preview. ### Source excerpt In this newsletter, we cover version 9.5 of Elasticsearch, the latest blogs and videos, and upcoming events. ## MindTopo reveals VLMs' spatial reasoning abilities DevFeed: [MindTopo reveals VLMs' spatial reasoning abilities]() Original publisher: [Read original article]() Author: Yunfei Ge, Anbang Liu, Qineng Wang, Johnalbert Garnica, Zihan Wang, Reuben Tan, Jianfeng Gao, Ruohan Zhang, Yining Hong, Jiajun Wu, Manling Li Published: 2026-08-12T16:00:00Z Content type: article Language: en Sources: [Microsoft Research]() Topics: [Benchmark](), [LLM evaluation / benchmarking](), [multimodal-ai](), [Language models](), [Robotics]() Tags: [ai](), [benchmark](), [large-language-models](), [multimodal](), [reasoning](), [research](), [research-blog](), [testing](), [vlms]() ### AI overview MindTopo is a benchmark for evaluating whether multimodal large language models can understand and manipulate topological relationships such as connectivity, enclosure, order, separation, and knots. It compares static recognition with interactive planning and finds that current models often lose track of structural relationships during sequences of actions. ### Source excerpt A path, a fence, a knot. MindTopo sets a new benchmark for testing how AI understands topological relationships and highlights new opportunities to strengthen spatial reasoning and planning. The post MindTopo reveals VLMs' spatial reasoning abilities appeared first on Microsoft Research. ## Building a Local, Multimodal AI Terminal Agent with Gemma 4 DevFeed: [Building a Local, Multimodal AI Terminal Agent with Gemma 4]() Original publisher: [Read original article]() Author: Arjun Prabhulal Published: 2026-08-12T09:25:11Z Content type: tutorial Language: en Sources: [Google Developer Experts - Medium]() Topics: [gemma4](), [Artificial Intelligence](), [multimodal-ai](), [Command-line interface](), [Large Language Model](), [Local AI](), [Code](), [Google]() Tags: [ai](), [ai-agent](), [artificial-intelligence](), [code](), [function-calling](), [gemma](), [gemma-4](), [google](), [google-cloud-platform](), [local-ai](), [multimodal](), [ollama](), [open-source](), [terminal]() ### AI overview A tutorial introduces Gemma 4 and builds a local multimodal terminal agent named gemma4-agent. It covers function calling, tool orchestration, text, image, and voice processing, plus Gemma 4's model variants and architecture. ### Source excerpt Introduction Open-source LLM models have been improving rapidly with tool calling, extended context windows, and native vision and audio capabilities, all while delivering strong benchmark performance. Gemma 4, recently introduced by Google Deepmind brings all of these features together in sizes efficient enough to run locally. In this article, we'll look at the capabilities of Gemma 4 and build a multimodal (Text, Vision, Voice) CLI agent (gemma4-agent) with function-calling capabilities. By the end, you'll have an agent that can chat, write, execute code, analyze images, and process voice instructions to deliver highly grounded responses. What is Gemma 4 Model ? Gemma 4 is Google DeepMind's open model family, released in April 2026 under the Apache 2.0 license. Built from the same research and technology behind Gemini 3, Gemma 4 is designed for high-performance reasoning, coding, multimodal understanding, and local AI execution across different model sizes. Features of Gemma 4 Models Improved Tool calling : Native function calling and tool orchestration, letting agents act autonomously without bloating prompt instructions Thinking mode : Built-in step-by-step thinking mode via the <|think|> token for complex multi-turn logic Context Windows : Up to 256K tokens on the 12B and larger models (128K on the edge-sized E2B/E4B) for processing long document and tool outputs Extended Multimodality : Gemma 4 models can process text,voice and images simultaneously like extracting data from charts, analyzing screenshots , and reviewing UI mockups. Gemma 4 Model Variants & SpecificationsGemma 4 Architecture Gemma 4 comes in five model sizes built around four architectural variants, each making different trade-offs between performance, inference speed, compute, and memory. Gemma4 Unified 12B vs Effective Parameters Effective-parameter models (E2B and E4B) are dense transformer models optimized for edge and on-device deployment. The "E" stands for effective parameters use Per-La ## Advancing AMIE towards expert-level audio-visual clinical consultations DevFeed: [Advancing AMIE towards expert-level audio-visual clinical consultations]() Original publisher: [Read original article]() Published: 2026-08-11T17:04:00Z Content type: article Language: en Sources: [The latest research from Google]() Topics: [Artificial Intelligence](), [Google](), [real-time]() Tags: [ai](), [audio](), [communication](), [health-bioscience](), [machine-intelligence](), [multimodal](), [partnership](), [reasoning](), [research](), [video]() ### AI overview Google Research describes an advance to AMIE, a medical AI system designed for real-time video consultations. In simulated consultations, AMIE demonstrated expert-level performance in a randomized controlled study while integrating spoken history with visual and auditory clinical cues. ### Source excerpt Health & Bioscience ## Meta is back with Muse Glimmer: local, agentic, multimodal, and open source DevFeed: [Meta is back with Muse Glimmer: local, agentic, multimodal, and open source]() Original publisher: [Read original article]() Author: Pedro Cuenca; merve; ben burtenshaw; Aritra Roy Gosthipaty Published: 2026-08-10T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [vlm](), [multimodal](), [Artificial Intelligence](), [hugging face](), [Benchmark](), [coding]() Tags: [agentic](), [artificial-intelligence](), [benchmark](), [hub](), [hugging-face](), [images](), [llama-cpp](), [llms](), [local](), [meta](), [model](), [multimodal](), [muse](), [open-source](), [videos](), [vllm](), [vlm](), [vlms]() ### AI overview Hugging Face presents Muse Glimmer, a local, agentic, multimodal, open-source 30B-parameter vision-language model developed with Meta. The article outlines its vision and language architecture, benchmark context, optional speculative decoding for faster generation, and support for both images and videos. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. [Next page]()