# Inference The process of making predictions by applying a trained machine-learning model to unlabeled examples. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Intern 2 offers an isolated environment and cloud inference for running personal bots DevFeed: [Intern 2 offers an isolated environment and cloud inference for running personal bots]() Original publisher: [Read original article]() Author: Thomas Claburn Published: 2026-09-16T23:28:59Z Content type: news Language: en Sources: [www.theregister.com - Articles]() Topics: [Bots](), [Mac Mini](), [Inference]() Tags: [ai-agents](), [ai-and-ml](), [automate](), [autonomous-ai](), [bots](), [edge-computing](), [inference](), [intern-2](), [mac-mini](), [open-source](), [personal-tech]() ### AI overview Intern 2 provides an isolated environment and cloud inference for automating tasks with personal bots. ### Source excerpt 'Intern 2' offers an isolated environment and cloudy inference to automate your world ## University of Manchester Uses NVIDIA Earth-2 to Forecast Air Pollution Across the UK DevFeed: [University of Manchester Uses NVIDIA Earth-2 to Forecast Air Pollution Across the UK]() Original publisher: [Read original article]() Author: Isha Salian Published: 2026-09-16T05:00:42Z Content type: article Language: en Sources: [NVIDIA Blog]() Topics: [Artificial Intelligence](), [AI Models](), [Supercomputing](), [DGX Spark](), [Simulation](), [Training AI Models](), [data](), [Inference](), [real-time]() Tags: [ai](), [ai-for-good](), [ai-models](), [climate](), [compute](), [dgx-spark](), [government](), [healthcare](), [inference](), [nvidia](), [open-source](), [real-time](), [science](), [supercomputing](), [training](), [uk]() ### AI overview The University of Manchester is working with NVIDIA to use Earth-2 generative AI models to forecast air pollution across the U.K. The team trained Earth-2 CorrDiff on chemistry-climate simulation data using Isambard-AI, added StormCast for time-dependent forecasts using air-quality observations, and demonstrated workflows on DGX Spark. ### Source excerpt Air pollution is a serious public health risk, contributing to an estimated 30,000 deaths in the U.K. alone last year. Data-driven insights can help -- but computing air quality with traditional chemistry-based models is expensive, which limits how detailed they can be and how regularly they can be run. David Topping, a professor in the [...] ## ML based ranking using Nrtsearch DevFeed: [ML based ranking using Nrtsearch]() Original publisher: [Read original article]() Author: Mohammad Mohtasham (Software Engineer); Tao Yu (Software Engineer) Published: 2026-09-16T00:00:00Z Content type: article Language: en Sources: [Yelp]() Topics: [Inference](), [Scalability](), [Latency](), [recommendation systems](), [Open Source]() Tags: [bridge](), [inference](), [machine-learning](), [overhead](), [performance](), [pipeline](), [scalability](), [service]() ### AI overview Yelp's Nrtsearch Inference Plugin embeds machine-learning ranking directly in the search layer. The article explains the scoring workflow, including model configuration, feature extraction, candidate ranking, and application-specific business logic. It describes how co-locating feature storage and inference reduces network transfer, serialization overhead, and latency compared with a standalone inference service. ### Source excerpt We've extended Nrtsearch with the Inference Plugin, which embeds ML-based ranking directly in the search layer -- eliminating the need for a standalone scoring service. We use Nrtsearch (read more information on the blog post), a Lucene-based open-source search engine built by Yelp, to power a variety of applications such as business search, reviews search, ad delivery and photo search. In this blog post, we give a high-level overview of the Machine Learning (ML) based scoring workflow in Nrtsearch. We'll show how ML models are configured and loaded, and how different applications use custom business logic to develop, test, and... ## Seagate and WD AI Storage Research Finds Enterprises Rank Storage Above Compute as the AI Bottleneck DevFeed: [Seagate and WD AI Storage Research Finds Enterprises Rank Storage Above Compute as the AI Bottleneck]() Original publisher: [Read original article]() Author: Lyle Smith Published: 2026-09-15T17:23:54Z Content type: news Language: en Sources: [StorageReview.com]() Topics: [Data Infrastructure](), [AI Infrastructure](), [idc](), [Inference](), [Retrieval-Augmented Generation](), [synthetic-data]() Tags: [ai](), [compute](), [data-infrastructure](), [datasets](), [enterprise](), [genai](), [hdd](), [idc](), [inference](), [reports](), [research](), [retrieval](), [storage](), [synthetic-data]() ### AI overview Seagate and WD published separate studies indicating that AI is increasing enterprise storage requirements and extending data retention. Although their headline percentages differ because they asked different questions, both reports point to storage becoming a larger part of AI infrastructure planning alongside growing archive and retrieval needs. ### Source excerpt Seagate and WD published separate AI storage studies within days of each other; the headline numbers: Seagate says 99% of enterprises expect AI to increase their storage requirements over the next three years, while WD's IDC research puts the comparable figure at 74%. Read the fine print, and both reports land in the same directional The post Seagate and WD AI Storage Research Finds Enterprises Rank Storage Above Compute as the AI Bottleneck appeared first on StorageReview.com. ## From Megawatts to Tokens: How NVIDIA Maximizes AI Factory Production DevFeed: [From Megawatts to Tokens: How NVIDIA Maximizes AI Factory Production]() Original publisher: [Read original article]() Author: Vishal Ganeriwala Published: 2026-09-15T16:55:59Z Content type: article Language: en Sources: [NVIDIA Blog]() Topics: [AI Factory](), [NVIDIA DSX](), [datacenter](), [Orchestration](), [Inference](), [High-Performance Computing]() Tags: [ai](), [ai-factory](), [ai-infrastructure](), [capacity](), [compute](), [dsx](), [efficiency](), [inference](), [nvidia](), [orchestration](), [production]() ### AI overview The article describes how Emerald AI's Conductor platform responds to utility demand signals by adjusting flexible data-center workloads while keeping high-priority AI inference running. It also reports that Lambda's validation found a fixed power budget could support 24% more token throughput when managed intelligently. ### Source excerpt On a sweltering August evening in Silicon Valley, as the sun dropped and air conditioning loads spiked, Silicon Valley Power sent a signal to an AI factory to adjust its power consumption. Varun Sivaram was watching on Zoom with about forty others -- his team at Emerald AI in their San Francisco conference room, engineers [...] ## How NVIDIA Groq 3 LPX Deterministic Execution Drives Power-Efficient High-Interactivity Inference on NVIDIA Vera Rubin DevFeed: [How NVIDIA Groq 3 LPX Deterministic Execution Drives Power-Efficient High-Interactivity Inference on NVIDIA Vera Rubin]() Original publisher: [Read original article]() Author: Tanya Lenz Published: 2026-09-15T16:55:00Z Content type: article Language: en Sources: [NVIDIA Developer](), [NVIDIA Technical Blog]() Topics: [Groq 3 LPX](), [LPX](), [NVIDIA Vera Rubin](), [Vera Rubin NVL72](), [groq](), [Inference](), [Low Latency](), [long-context]() Tags: [agentic-ai-generative-ai](), [ai-factory](), [ai-inference](), [ai-platform](), [drive](), [dsx](), [groq](), [groq-3-lpx](), [inference](), [inference-performance](), [latency](), [low-latency](), [lpx](), [nvidia-vera-rubin](), [nvl72](), [performance](), [power-management](), [vera-rubin](), [vera-rubin-nvl72]() ### AI overview This NVIDIA developer article explains how Groq 3 LPX uses deterministic execution across 256 LPU chips to support low-latency inference on NVIDIA Vera Rubin. It describes compiler-scheduled execution and power-management techniques including Preemptive Power and Clock Period Synthesis. ### Source excerpt Power is a defining constraint for AI factories. As AI workloads demand a full compute platform to serve them, each component of that platform must maximize... ## Build an AI-powered product tagging system with Amazon SageMaker serverless model customization DevFeed: [Build an AI-powered product tagging system with Amazon SageMaker serverless model customization]() Original publisher: [Read original article]() Author: Linpo Guo Published: 2026-09-15T16:11:36Z Content type: tutorial Language: en Sources: [Artificial Intelligence]() Topics: [Artificial Intelligence](), [Amazon SageMaker AI](), [Fine-tuning](), [rlvr](), [grpo](), [Inference](), [model-deployment](), [Reinforcement learning](), [SDK](), [Serverless]() Tags: [ai](), [amazon-sagemaker](), [amazon-sagemaker-ai](), [aws](), [compute](), [cost](), [customization](), [expert-400](), [fine-tuning](), [inference](), [jobs](), [reinforcement-learning](), [rlvr](), [sdk](), [technical-how-to]() ### AI overview This walkthrough shows how to build a product tagging system by customizing Qwen3-8B with supervised fine-tuning and reinforcement learning with verifiable rewards on Amazon SageMaker serverless model customization. It then deploys the optimized model for asynchronous inference to enrich retail catalogs. ### Source excerpt Manually tagging thousands of catalog products is slow and inconsistent. This walkthrough shows how to customize Qwen3-8B with supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR) on Amazon SageMaker serverless model customization, then deploy it for asynchronous inference to build a cost-efficient product tagging system. ## Axelera Europa Ships: 629 TOPS at 45W Per AIPU, in Validated Dell XE5 and Supermicro Servers DevFeed: [Axelera Europa Ships: 629 TOPS at 45W Per AIPU, in Validated Dell XE5 and Supermicro Servers]() Original publisher: [Read original article]() Author: Harold Fritts Published: 2026-09-15T13:00:00Z Content type: news Language: en Sources: [StorageReview.com]() Topics: [Artificial Intelligence](), [Inference](), [Computer vision](), [servers](), [dell](), [RISC-V]() Tags: [ai](), [computer-vision](), [dell](), [enterprise](), [inference](), [risc-v](), [servers]() ### AI overview Axelera AI is shipping Europa, a second-generation AI Processing Unit, in bare-chip and PCIe card configurations. The company says the 45W device delivers 629 TOPS and supports on-premises inference workloads including generative AI, vision-language models, and computer vision. The Edge 232p card is shipping in validated Dell XE5 and Supermicro 111AD systems. ### Source excerpt Axelera AI is shipping Europa, the second-generation AI Processing Unit (AIPU) it has been previewing since last year, and it's launching with validated servers from Dell and Supermicro attached. The Eindhoven company's pitch is inference on infrastructure the customer controls: agentic systems, vision-language models, generative AI, and computer vision running in a standard rackmount server The post Axelera Europa Ships: 629 TOPS at 45W Per AIPU, in Validated Dell XE5 and Supermicro Servers appeared first on StorageReview.com. ## Lightbits Inferra KV Cache Engine Claims 16x Session Density and 10M-Token Contexts DevFeed: [Lightbits Inferra KV Cache Engine Claims 16x Session Density and 10M-Token Contexts]() Original publisher: [Read original article]() Author: Harold Fritts Published: 2026-09-14T16:23:21Z Content type: news Language: en Sources: [StorageReview.com]() Topics: [Inference](), [Inference Performance](), [Orchestration](), [GPU](), [Multi-tenancy](), [sglang](), [TensorRT](), [vllm](), [Security]() Tags: [ai](), [cache](), [concurrent](), [enterprise](), [gpu](), [inference](), [latency](), [nvme](), [orchestration](), [security]() ### AI overview Lightbits Labs is introducing Inferra, a KV cache orchestration engine for AI inference. It virtualizes GPU memory across DRAM and NVMe storage, preserving attention states for long-context and multi-session workloads. Lightbits claims up to 16 times more concurrent sessions, more than 100 times lower latency than recomputation, and context windows of up to 10 million tokens. Inferra supports vLLM, TensorRT, and SGLang and includes tiering, predictive prefetching, tenant isolation, and encrypted data transfer. ### Source excerpt Lightbits Labs, the company that invented NVMe over TCP, is moving into inference software with Inferra, a KV cache orchestration engine that makes its public debut tomorrow, September 15, at the AI Infra Summit in Santa Clara. The software virtualizes GPU memory across DRAM and NVMe storage tiers and turns the KV cache into a The post Lightbits Inferra KV Cache Engine Claims 16x Session Density and 10M-Token Contexts appeared first on StorageReview.com. ## ๐Ÿ”๐Ÿง  Pinterest's Fix for the Hardest Problem in ML Infra DevFeed: [๐Ÿ”๐Ÿง  Pinterest's Fix for the Hardest Problem in ML Infra]() Original publisher: [Read original article]() Author: Alexandre Zajac Published: 2026-09-14T15:31:30Z Content type: article Language: en Sources: [Hungry Minds]() Topics: [recommendation systems](), [Feature Engineering](), [Inference](), [Streaming](), [configuration](), [Framework](), [data](), [Low Latency]() Tags: [batch](), [configuration](), [data](), [feature-engineering](), [low-latency](), [ml](), [pinterest](), [ranking](), [recommendation-systems](), [streaming]() ### AI overview Pinterest redesigned its user-sequence platform for ranking, retrieval, and recommendation systems by defining signals once and instantiating them consistently across streaming, batch, and serving workloads. The approach uses Python configuration with validated schemas, a shared execution engine, cooperating streaming and batch paths, and columnar time-partitioned storage to improve freshness, completeness, consistency, and operational efficiency. ### Source excerpt PLUS: OpenAI agents beat math ๐Ÿงฎ, Test techniques for agents โšก, Postgres survival guide ๐Ÿ“– ## On-Device AI Series (Part 5): LiteRT-LM DevFeed: [On-Device AI Series (Part 5): LiteRT-LM]() Original publisher: [Read original article]() Author: OฤŸuzhan Aslan Published: 2026-09-14T05:59:12Z Content type: tutorial Language: en Sources: [ProAndroidDev - Medium]() Topics: [LiteRT](), [On-device AI](), [Large Language Model](), [Inference](), [Streaming](), [GPU](), [LLM evaluation / benchmarking](), [PyTorch]() Tags: [ai](), [android](), [android-development](), [benchmarking](), [gpu](), [inference](), [litert](), [llama](), [llm](), [multimodal](), [on-device](), [on-device-ai](), [programming](), [pytorch](), [qwen](), [software-development](), [streaming]() ### AI overview This tutorial explains LiteRT-LM for running large language models on-device. It covers the Engine/Session API, streaming output, system prompts, tool calling, multimodal inputs, thinking mode, and CPU-versus-GPU benchmarking. The article also discusses tradeoffs involving privacy, network independence, latency, memory, sampling configuration, and model capability compared with cloud APIs. ### Source excerpt Put your phone in airplane mode. Open the app, type a question, and watch the answer arrive one token at a time -- no spinner waiting on a network round-trip, no API key, no per-token bill, and nothing you typed ever leaving the device. LiteRT-LM removes the genuinely hard parts of running an LLM on-device -- KV-cache management, token streaming, backend selection -- but it doesn't remove your job so much as relocate it. What's left on your plate is a short, specific list: sizing a combined input+output token budget, owning your own sampling defaults, hand-building system prompts and tool calling out of raw text, and one native-library collision that presents as a SIGSEGV rather than a build error. Know those going in and the API itself is a clean three-step pattern. We'll get there in that order: Why you'd choose this runtime and what it costs you versus the cloud. The Engine/Session model you need to read the code at all. Real implementation samples -- streaming, system prompts and tool calling, multimodal inputs, thinking mode, and CPU-vs-GPU benchmarking. The anti-patterns to avoid. A developer-friendliness rating on the same rubric as Parts 1-4. Why Use LiteRT-LM? You reach for LiteRT-LM instead of hand-rolling generation on top of raw LiteRT when: You need multi-turn conversation, not single-shot inference -- session state and KV-cache bookkeeping are handled for you, and resetting a conversation is a session swap, not a model reload. You need streaming output -- token-by-token delivery for a responsive chat UI, instead of a blocking call that returns everything at once. You're choosing between CPU and GPU per device -- the explicit backend parameter turns that into a runtime decision instead of a build-time guess. You want a pre-converted model without doing your own PyTorch-to-LiteRT conversion work -- the Model Zoo covers Gemma, Qwen, Llama, and more out of the box. You're willing to own sampling -- the engine won't pick sane decoding defaults for you; that's on the ## "Machine translation is still broken for most of the world's languages": Cohere builds non-reasoning for a reason DevFeed: ["Machine translation is still broken for most of the world's languages": Cohere builds non-reasoning for a reason]() Original publisher: [Read original article]() Author: Adrian Bridgwater Published: 2026-09-13T14:21:46Z Content type: news Language: en Sources: [The New Stack]() Topics: [Mixture of Experts (MoE)](), [cohere](), [Artificial Intelligence](), [Benchmark](), [Inference](), [qwen](), [gemma4](), [Google]() Tags: [ai](), [ai-infrastructure](), [ai-models](), [aya](), [benchmark](), [cohere](), [gemma](), [google](), [inference](), [large-language-models](), [mixture-of-experts-moe](), [model](), [open](), [qwen]() ### AI overview Cohere's North Small Translate is an open-weight mixture-of-experts machine translation model covering 50 languages. The article discusses its non-reasoning design, sovereign AI positioning, deployment options, efficiency claims, and reported WMT26 benchmark comparisons. ### Source excerpt Enterprise AI company Cohere announced North Small Translate last week, a mixture-of-experts (MOE) open-weight machine translation model that works across The post "Machine translation is still broken for most of the world's languages": Cohere builds non-reasoning for a reason appeared first on The New Stack. ## Enterprise-Grade Precision for Long-Context Multimodal Embedding Inference on Cloud TPU DevFeed: [Enterprise-Grade Precision for Long-Context Multimodal Embedding Inference on Cloud TPU]() Original publisher: [Read original article]() Author: Anthony Su; Injae Kwak Published: 2026-09-12T11:04:33.891311Z Content type: article Language: en Sources: [Google Developers Blog]() Topics: [Embeddings](), [vllm](), [Inference](), [long-context](), [autoscaling](), [Kubernetes](), [Cloud](), [Google](), [Artificial Intelligence](), [model-deployment](), [Open Source]() Tags: [autoscaling](), [embedding](), [github](), [gpu](), [inference](), [kubernetes](), [long-context](), [multimodal](), [open-source](), [retrieval](), [tpu](), [vllm]() ### AI overview This article explains how Google Cloud integrated native TPU support into vLLM to serve long-context, multimodal embedding models at production scale. It describes elastic TPU provisioning with Google Kubernetes Engine, autoscaling across accelerator types, and TPU-specific optimizations for long sequences and chunked prefill. The resulting setup is designed to preserve numerical parity with GPU reference baselines while supporting high-throughput semantic retrieval applications. ### Source excerpt Google Cloud has natively integrated TPU support into the vLLM serving engine, allowing developers to elastically scale high-demand embedding pipelines using Google Kubernetes Engine (GKE). To handle massive 15K+ token contexts for models like Qwen3-Embedding-8B, the engineering team implemented TPU-specific optimizations such as hardware-safe tensor alignment, JAX/XLA compilation pre-warming, and a hybrid StepPool architecture for chunked prefill management. These enhancements achieve near-perfect numerical parity with reference GPU baselines, and developers can immediately leverage the open-sourced setup recipes on the AI-Hypercomputer GitHub to build their own high-throughput semantic retrieval applications. ## OpenAI's researchers burned $7,000 a day on AI agents -- now it's opening the floodgates DevFeed: [OpenAI's researchers burned $7,000 a day on AI agents -- now it's opening the floodgates]() Original publisher: [Read original article]() Author: Amanda Caswell Published: 2026-09-11T21:27:42Z Content type: news Language: en Sources: [The New Stack]() Topics: [Inference](), [long-context]() Tags: [agents](), [ai](), [ai-agents](), [ai-infrastructure](), [api](), [cloud-services](), [codex](), [compute](), [developers](), [inference](), [openai](), [research](), [sandbox](), [tools]() ### AI overview OpenAI's public-beta Agents API lets developers run long-lived agents with managed job state, context compression, optional tools, parallel subagents, and execution in OpenAI's sandbox or developer-controlled infrastructure. The article highlights the resulting inference and compute costs, citing internal research-agent usage figures. ### Source excerpt OpenAI rolled out its Agents API in public beta Thursday, opening the backend behind Codex to developers looking to run The post OpenAI's researchers burned $7,000 a day on AI agents -- now it's opening the floodgates appeared first on The New Stack. ## Cohere's new translation model is open weights -- but not for commercial use DevFeed: [Cohere's new translation model is open weights -- but not for commercial use]() Original publisher: [Read original article]() Author: Meredith Shubel Published: 2026-09-11T17:50:11Z Content type: news Language: en Sources: [The New Stack]() Topics: [Inference](), [moe]() Tags: [ai](), [ai-infrastructure](), [ai-models](), [ai-strategy](), [api](), [cohere](), [hugging-face](), [inference](), [mixture-of-experts](), [model](), [models](), [open](), [production](), [release]() ### AI overview Cohere released North Small Translate 1.0 as open weights under CC BY-NC 4.0, allowing download, evaluation, and study but requiring a commercial agreement for production use. Commercial deployment requires a license and use of Cohere's managed Model Vault platform. ### Source excerpt This week, Cohere released North Small Translate 1.0 under a CC BY-NC 4.0 license: the weights are there to download, The post Cohere's new translation model is open weights -- but not for commercial use appeared first on The New Stack. ## NVIDIA Personal AI Router Distributes AI Tasks across Local Compute DevFeed: [NVIDIA Personal AI Router Distributes AI Tasks across Local Compute]() Original publisher: [Read original article]() Author: Sergio De Simone Published: 2026-09-11T15:00:00Z Content type: news Language: en Sources: [InfoQ]() Topics: [Local AI](), [Inference](), [GPU](), [Hardware]() Tags: [agent](), [agents](), [ai](), [ai-ml-data-engineering](), [architecture-design](), [compute](), [demo](), [development](), [gpu](), [inference](), [large-language-models](), [local](), [ml-data-engineering](), [news](), [node](), [nvidia](), [nvidia-pair-ai-task-router](), [open-source](), [orchestration](), [performance]() ### AI overview NVIDIA has introduced PAIR in beta, a local router that distributes inference requests across compatible computers for multi-agent AI workloads. It works with local inference services such as Ollama and LM Studio and selects a node based on model and engine requirements. ### Source excerpt NVIDIA Personal AI Router (PAIR), now available in beta, lets you combine the inference capacity of multiple computers on your local network and automatically distribute AI requests among them. It is primarily designed for local multi-agent AI workloads, where multiple independent model calls can otherwise overwhelm one GPU. By Sergio De Simone ## Building Pinterest's VLM Serving Stack on NVIDIA Dynamo DevFeed: [Building Pinterest's VLM Serving Stack on NVIDIA Dynamo]() Original publisher: [Read original article]() Author: Pinterest Engineering Published: 2026-09-10T23:08:16Z Content type: article Language: en Sources: [Pinterest Engineering Blog - Medium]() Topics: [vlm](), [Inference](), [model-deployment](), [Orchestration](), [Latency](), [Caching](), [Open Source Models & Datasets]() Tags: [blackwell](), [cache](), [dynamo](), [embeddings](), [engineering](), [inference](), [latency](), [llm](), [multimodal](), [multimodal-ai](), [nvidia](), [nvidia-blackwell](), [open-source](), [orchestration](), [performance](), [pinterest](), [vllm](), [vlm](), [vlm-serving]() ### AI overview Pinterest describes its VLM serving stack built on NVIDIA Blackwell GPUs and NVIDIA Dynamo. The stack addresses multimodal inference demands such as image processing, variable prefill costs, KV-cache pressure, routing, and cache offloading. ### Source excerpt Lei Pan | Senior Software Engineer; Salina Wu | Senior Software Engineer; Cristian Lopez | Software Engineer I; Guangtong Bai | Staff Software Engineer; Soam Acharya | Principal Engineer; Saurabh Vishwas Joshi | Principal Engineer; Chia-Wei Chen | Staff Software Engineer; Ambud Sharma | Principal Engineer Why VLM Serving Matters at Pinterest Pinterest is a visual search and discovery platform, so its AI systems must reason over both language and visual content. Vision-language models (VLMs), which can interpret images, compare visual candidates, and respond naturally to user intent, are becoming the foundation for the next generation of Pinterest experiences: Pinterest Assistant, hybrid search, multimodal reranking, content understanding, signal generation, content safety, and more. This direction also reflects Pinterest's broader strategy to customize open-source models to meet its product & scale needs. Pinterest Assistant is a standout example. This multi-turn conversational experience covers both user language and visual content. Serving it requires low-latency VLM inference over rich multimodal context as well as reworking Qwen3-VL with proprietary multimodal embeddings to cut runtime cost while improving performance. Serving VLMs, however, introduces more challenges compared to text-only LLM workloads. Requests may carry multiple images, require extra vision encoder computation, incur larger and more variable prefill cost, and create higher KV cache pressure. To support this new class of models & product experiences, we built Pinterest's VLM serving stack on top of NVIDIA Blackwell GPUs and NVIDIA Dynamo. Blackwell GPUs incorporate many architectural innovations that are uniquely positioned for today's most demanding AI workloads -- including higher BF16/FP8 compute throughput, increased memory bandwidth, and larger HBM memory capacity -- that enable dramatically higher performance for inference. Dynamo provides a distributed inference orchestration layer that g ## Reduce LLM latency with prefix-aware routing on Amazon SageMaker Inference DevFeed: [Reduce LLM latency with prefix-aware routing on Amazon SageMaker Inference]() Original publisher: [Read original article]() Author: Kareem Syed-Mohammed Published: 2026-09-10T21:58:09Z Content type: release Language: en Sources: [Artificial Intelligence]() Topics: [Large Language Model](), [Low-Latency Inference](), [Caching](), [Inference](), [Inference Performance]() Tags: [advanced-300](), [amazon-sagemaker](), [amazon-sagemaker-ai](), [announcements](), [benchmarks](), [caching](), [inference](), [latency](), [llama](), [llm](), [routing](), [tensorrt-llm](), [vllm]() ### AI overview Amazon SageMaker Inference introduces prefix-aware routing for LLM requests. By consistently sending requests with matching prompt prefixes to the same instance, it improves reuse of cached KV computations and can reduce time to first token. ### Source excerpt Amazon SageMaker Inference now offers prefix-aware routing, a routing strategy that sends requests sharing the same prompt prefix to the same instance so the KV cache stays warm. In benchmarks on Llama 3.1 70B, it reduced P50 time-to-first-token by up to 77% and raised KV cache hit rates from about 25% to over 80%. ## Reduce inference cold starts on Amazon SageMaker HyperPod with model caching DevFeed: [Reduce inference cold starts on Amazon SageMaker HyperPod with model caching]() Original publisher: [Read original article]() Author: Kareem Syed-Mohammed Published: 2026-09-10T21:37:49Z Content type: tutorial Language: en Sources: [Artificial Intelligence]() Topics: [Caching](), [Inference](), [model-deployment](), [Large Language Model](), [autoscaling](), [Kubernetes](), [Amazon S3]() Tags: [amazon](), [amazon-s3](), [amazon-sagemaker](), [amazon-sagemaker-hyperpod](), [announcements](), [autoscaling](), [benchmark](), [caching](), [expert-400](), [inference](), [kubernetes](), [llm]() ### AI overview Amazon SageMaker HyperPod model caching preloads model weights and container images onto cluster nodes, reducing inference-pod cold starts by serving assets from local NVMe storage. ### Source excerpt Amazon SageMaker HyperPod now supports model caching for inference, which pre-loads model weights and container images onto cluster nodes so pods read from local NVMe storage instead of downloading over the network. Learn how model caching cuts cold starts from tens of minutes to seconds, how it works, and how to enable it. ## Caching Responses for Repeated Questions in Production LLM Applications DevFeed: [Caching Responses for Repeated Questions in Production LLM Applications]() Original publisher: [Read original article]() Author: Avi Chawla Published: 2026-09-10T21:34:26Z Content type: article Language: en Sources: [Daily Dose of Data Science]() Topics: [Caching](), [Redis](), [Embeddings](), [Large Language Model](), [Inference]() Tags: [cache](), [caching](), [embeddings](), [inference](), [llm](), [redis]() ### AI overview Production LLM applications can reduce repeated model calls by caching generated responses and matching new questions to prior questions using embeddings. The article discusses Redis LangCache, including similarity thresholds, expiration, data isolation, monitoring, and an example where a cache hit was roughly 6x faster than direct inference. ### Source excerpt What production agents must preserve between turns, and what they must reset. ## How Uber and Other Companies Are Reducing AI Costs with Open Models and Model Routing DevFeed: [How Uber and Other Companies Are Reducing AI Costs with Open Models and Model Routing]() Original publisher: [Read original article]() Author: Ivan Klaric Published: 2026-09-10T17:22:31Z Content type: article Language: en Sources: [The Pragmatic Engineer]() Topics: [Model Routing](), [Artificial Intelligence](), [Inference](), [LLM evaluation / benchmarking](), [Benchmark]() Tags: [ai](), [benchmark](), [benchmarking](), [cost](), [inference](), [model-routing](), [models](), [optimize]() ### AI overview The article reports that companies including Uber, Pinterest, Stripe, Coinbase, Ramp, and AT&T are reducing AI costs by using open models, model routing, benchmarking, and other optimization techniques. It details Uber's use of cheaper inference services, optimized model selection, smaller subagent models, prompt caching, and automatic compaction. ### Source excerpt Uber, Pinterest, Stripe, Coinbase, Ramp, and AT&T are making large savings on their AI bills by dropping proprietary models and using smart model routing. ## Cloudera and Mistral Partner to Bring Specialized, Sovereign Intelligence to Enterprise Data DevFeed: [Cloudera and Mistral Partner to Bring Specialized, Sovereign Intelligence to Enterprise Data]() Original publisher: [Read original article]() Published: 2026-09-10T10:42:55Z Content type: news Language: en Sources: [Mistral AI Blog]() Topics: [model-deployment](), [Inference](), [data](), [datasets]() Tags: [ai](), [ai-models](), [cloud](), [data](), [enterprise](), [inference](), [on-prem](), [partnership](), [train]() ### AI overview Cloudera and Mistral announce a partnership to deploy and train customized AI models on enterprise data across hybrid, on-premises, cloud, and air-gapped environments while retaining data control. ### Source excerpt Cloudera and Mistral join forces to bring specialized, sovereign AI intelligence to enterprise data, helping regulated industries innovate on their own terms. ## Creating an AI Platform for classic ML online inference DevFeed: [Creating an AI Platform for classic ML online inference]() Original publisher: [Read original article]() Author: Rohith Leeladharan Published: 2026-09-10T07:26:46Z Content type: tutorial Language: en Sources: [Amex GBT Technology]() Topics: [AI Platform](), [Inference](), [MLOps](), [Machine learning](), [model-deployment]() Tags: [ai](), [ai-platform](), [ai-platform-engineering](), [deploy](), [experiments](), [feature-store](), [inference](), [machine-learning](), [ml](), [mlops](), [monitoring](), [predictions]() ### AI overview This article describes how American Express Global Business Travel built an AI platform for deploying classic machine-learning systems and supporting online inference. It explains the platform's requirements--simplicity, self-service, experimentation, and continuous improvement--and details the pre-process, predict, post-process pattern used by inference engines. ### Source excerpt Introduction In 2021, we were given the mission to have AI Systems running in production. The team, instead of just following a classical MLOps process, that involves transforming a Jupyter notebook into a product running in production, decided to go further by creating a platform to deploy AI systems in production. The team decided the platform should respect these requirements: Simplicity: The code powering AI systems should be simple, readable, and easy to maintain -- less intricacy means fewer bugs in production and greater reliability. Self-service: Anyone should be able to build and deploy AI systems autonomously, without depending on a central team. Experimentation: The platform should make it easy to run and iterate on experiments. Continuous improvement: Data related to events and interactions within AI systems must be captured, enabling monitoring and continuous improvement over time. In this article, we will walk through the work done to build a platform that fulfills these four requirements. Background At American Express Global Business Travel, we use machine learning (ML) models for a variety of user experiences like ranking hotel and flight search results. Our ML models are wrapped in inference engines that handle both pre-processing of input data before we run a prediction with the model, and post-processing of output data before returning the output to the caller. The overall flow looks something like this: Figure 1: Handling an inference request A client service that would like the ML model's predictions provides necessary context about the request like which user the request is for. Then, optionally, the inference engine fetches any necessary features for inference from our feature store [part 1][part 2]. Finally, it pre-processes the data, runs the predictions using the trained ML model, and does any necessary post-processing of the model output before returning the response to the caller. We call this the pre-process, predict, post-process patter ## Rebuilding AUTOMATIC1111 with Gradio Workflow DevFeed: [Rebuilding AUTOMATIC1111 with Gradio Workflow]() Original publisher: [Read original article]() Author: yuvraj sharma; Abubakar Abid Published: 2026-09-10T00:00:00Z Content type: tutorial Language: en Sources: [Hugging Face - Blog]() Topics: [AI Development](), [Graphs](), [stable-diffusion](), [Inference](), [Large Language Model](), [hugging face](), [vlm]() Tags: [automatic1111](), [comfyui](), [diffusion](), [flux](), [gradio](), [hugging-face](), [image-to-image](), [image-to-video](), [inference](), [inference-providers](), [llm](), [models](), [python](), [space](), [spaces](), [text-to-image](), [vlm](), [workflow](), [workflows]() ### AI overview A walkthrough of Workflow1111, a Gradio graph that recreates AUTOMATIC1111-style media pipelines with connected operator nodes for image generation, editing, prompting, and related tasks. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. [Next page]()