# Caching A computing technique that stores reusable results or response messages so later equivalent requests can be served with lower latency and network overhead. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Implement a correctness-safe Bloom filter lookup with Amazon ElastiCache for Valkey and Amazon Aurora PostgreSQL DevFeed: [Implement a correctness-safe Bloom filter lookup with Amazon ElastiCache for Valkey and Amazon Aurora PostgreSQL]() Original publisher: [Read original article]() Author: Chintan Agrawal Published: 2026-09-17T16:01:43Z Content type: tutorial Language: en Sources: [AWS Database Blog]() Topics: [bloom-filter](), [valkey](), [Amazon Aurora](), [Cache](), [PostgreSQL](), [Caching](), [consistency](), [Latency]() Tags: [advanced-300](), [amazon-aurora](), [amazon-elasticache](), [architecture](), [bloom-filter](), [cache](), [consistency](), [latency](), [performance](), [postgresql](), [postgresql-compatible](), [technical-how-to](), [valkey]() ### AI overview This post explains a three-tier membership lookup that combines a Bloom filter in Amazon ElastiCache for Valkey, an exact-match cache, and Amazon Aurora PostgreSQL as the relational source of truth. The design provides fast negative checks and preserves correctness when false positives would affect business outcomes. ### Source excerpt This post shows how to compose a Bloom filter with an exact-match cache and a relational source of truth into a three-tier, correctness-safe membership lookup using Amazon ElastiCache for Valkey and Amazon Aurora PostgreSQL, serving sub-millisecond decisions at peak throughput without false-positive risk. ## How Intuit and AWS systematically improved resiliency on ElastiCache using AWS Fault Injection Service DevFeed: [How Intuit and AWS systematically improved resiliency on ElastiCache using AWS Fault Injection Service]() Original publisher: [Read original article]() Author: Ryan Sheahan Published: 2026-09-17T15:59:44Z Content type: article Language: en Sources: [AWS Database Blog]() Topics: [Amazon ElastiCache](), [AWS Fault Injection Service (FIS)](), [Resilience](), [Caching](), [reliability](), [Availability]() Tags: [advanced-300](), [amazon-elasticache](), [applications](), [architecture](), [availability](), [aws](), [aws-fault-injection-service-fis](), [caching](), [clusters](), [customer-experience](), [customer-solutions](), [monitoring](), [recovery](), [resilience](), [resiliency]() ### AI overview AWS and Intuit describe validating Amazon ElastiCache resilience during an Availability Zone impairment with AWS Fault Injection Service. The post covers experiment setup, measurements, configuration issues, and reported recovery improvements under production-level load. ### Source excerpt Learn how Intuit and AWS validated Amazon ElastiCache resilience under a real Availability Zone impairment using AWS Fault Injection Service, cutting recovery from over 50 minutes to under 2 minutes with no manual intervention and reducing customer impact to effectively zero. ## Episode 70 "AOT Caching - Netflix' Practice vs OpenJDK's Theory" \[I/O\] DevFeed: [Episode 70 "AOT Caching - Netflix' Practice vs OpenJDK's Theory" \[I/O\]]() Original publisher: [Read original article]() Author: Nicolai Parlog Published: 2026-09-17T00:00:00Z Content type: article Language: en Sources: [Inside Java]() Topics: [Java](), [Caching](), [Project Leyden](), [openjdk](), [Netflix](), [IO]() Tags: [aot](), [caching](), [java](), [netflix](), [openjdk](), [performance](), [podcast](), [project-leyden]() ### AI overview Episode 70 of the Inside Java Podcast discusses ahead-of-time caching for Java applications. The episode covers how training runs populate a cache that can reduce JVM startup work in production, along with Netflix's canary-deployment approach for collecting training data and creating caches for later deployments. ### Source excerpt From time scales to checkpointing, from managing to inspecting cache files, there are a number of details to consider when creating AOT caches for faster Java application launches. ## Local and distributed cache coherence: stale data caused by missed invalidation messages DevFeed: [Local and distributed cache coherence: stale data caused by missed invalidation messages]() Original publisher: [Read original article]() Author: hello@ankit-rana.com Published: 2026-09-17T00:00:00Z Content type: opinion Language: en Sources: [Ankit Rana | Mechanical Sympathy]() Topics: [Caching](), [Redis](), [data](), [Network]() Tags: [cache](), [cache-coherence](), [caching](), [consistency](), [data](), [distributed-cache](), [distributed-systems](), [jvm](), [network](), [redis](), [ttl]() ### AI overview The article explains that adding an in-process cache in front of a distributed cache can improve latency but creates independently stale copies across service instances. It focuses on Redis pub/sub invalidation, which provides no persistence, acknowledgement, retry, or replay, allowing instances to miss invalidation messages and serve stale data until their TTL expires. ### Source excerpt An in-process cache in front of a distributed cache removes a network hop and adds one independent copy of the data per instance, each of which can be stale on its own schedule. Invalidation is normally broadcast over pub/sub, which is fire and forget, so any instance that is restarting, garbage collecting or briefly disconnected simply misses the message and serves stale data until its TTL expires. That TTL is not a performance setting, it is the maximum duration of incorrectness. ## Magic statics vs. std::call\_once DevFeed: [Magic statics vs. std::call\_once]() Original publisher: [Read original article]() Author: Raymond Chen Published: 2026-09-16T14:00:00Z Content type: article Language: en Sources: [Raymond Chen]() Topics: [Code](), [Caching](), [execution](), [function](), [class]() Tags: [cache](), [class](), [code](), [execution](), [function](), [old-new-thing](), [static]() ### AI overview The article compares function-local static variables with std::call_once for caching values that are expensive to calculate. It explains why magic statics are unsuitable for lazily initializing per-instance data and points to deferred execution as a related alternative. ### Source excerpt They sort of do the same thing, but differently. The post Magic statics vs. std::call_once appeared first on The Old New Thing. ## JavaScript bundling and HTTP performance trade-offs DevFeed: [JavaScript bundling and HTTP performance trade-offs]() Original publisher: [Read original article]() Author: Khan Academy Published: 2015-11-23T23:00:00Z Content type: article Language: en Sources: [Khan Academy]() Topics: [JavaScript](), [HTTP](), [Caching](), [browserify](), [Webpack]() Tags: [browserify](), [cache](), [engineering](), [http](), [infrastructure](), [js](), [news](), [webpack]() ### AI overview The article examines trade-offs between bundling JavaScript files, serving them separately, and using semantically related bundles. It explains how these approaches balance HTTP request overhead, browser fetch limits, cacheability, and unused downloaded code. ### Source excerpt By Craig Silverstein Bundling JavaScript files The traditional advice for web developers is to bundle the JavaScript files ... Read more ## Memcached-Backed Content Infrastructure DevFeed: [Memcached-Backed Content Infrastructure]() Original publisher: [Read original article]() Author: Khan Academy Published: 2017-05-15T22:00:00Z Content type: article Language: en Sources: [Khan Academy]() Topics: [Caching](), [Back end](), [data](), [Front end]() Tags: [backend](), [cache](), [engineering](), [infrastructure](), [memcached](), [news](), [profiling]() ### AI overview Khan Academy describes testing and rolling out a Memcached-backed content-serving backend. The article explains the existing content versioning and storage model, the performance limitations of fetching many individual items, and the prior approach of loading a compressed site-wide content bundle at server startup. ### Source excerpt By Ben Kraft Last post, I wrote about how we did profiling on App Engine's Memcached service to ... Read more ## How Khan Academy Scaled to 2.5x Traffic During the Coronavirus Pandemic DevFeed: [How Khan Academy Scaled to 2.5x Traffic During the Coronavirus Pandemic]() Original publisher: [Read original article]() Author: Khan Academy Published: 2020-05-09T22:00:00Z Content type: article Language: en Sources: [Khan Academy]() Topics: [Scalability](), [Architecture & Design](), [Serverless](), [Caching](), [Google Cloud Platform (GCP)](), [Load Balancing]() Tags: [architecture](), [caching](), [cdn](), [engineering](), [gcp](), [google-cloud](), [load-balancing](), [news](), [scalability](), [serverless](), [web]() ### AI overview Marta Kosarchyn explains how Khan Academy handled site usage reaching 2.5 times the previous year's level during the coronavirus pandemic. The article attributes the scalability to Google Cloud services, serverless infrastructure, Datastore, Memcache, Fastly CDN caching, and advance preparation. ### Source excerpt By Marta Kosarchyn Talk about rapid scaling... A few months ago I posted some thoughts on scaling and ... Read more ## Prompt Caching Support in Spring AI with Anthropic Claude DevFeed: [Prompt Caching Support in Spring AI with Anthropic Claude]() Original publisher: [Read original article]() Author: Stelios Anastasakis Published: 2026-09-16T07:49:56Z Content type: tutorial Language: en Sources: [Baeldung]() Topics: [Spring AI](), [Anthropic Claude](), [Caching]() Tags: [anthropic](), [anthropic-claude](), [artificial-intelligence](), [artificial-intelligence-anthropic-spring-ai-chatclient](), [caching](), [how-to](), [spring-ai](), [spring-ai-chatclient]() ### AI overview This tutorial explains how prompt caching works in Spring AI with Anthropic Claude. It covers dependencies, model-specific requirements and limitations, configuration options, caching hierarchy, and practical considerations. Prompt caching can reduce latency and input-token costs when prompt prefixes are reused. ### Source excerpt Learn how prompt caching works, the limitations for different Claude models, and how to use it in Spring AI. The post Prompt Caching Support in Spring AI with Anthropic Claude first appeared on Baeldung. ## Optimizing cost and latency with Amazon Bedrock prompt caching DevFeed: [Optimizing cost and latency with Amazon Bedrock prompt caching]() Original publisher: [Read original article]() Author: Daniel Abib Published: 2026-09-15T16:18:19Z Content type: tutorial Language: en Sources: [Artificial Intelligence]() Topics: [Amazon Bedrock](), [Caching](), [Latency](), [API](), [foundation-models](), [LangChain](), [Multi-tenancy]() Tags: [advanced-300](), [amazon-bedrock](), [api](), [caching](), [foundation-models](), [integration](), [langchain](), [latency](), [pricing](), [technical-how-to](), [tokens]() ### AI overview This AWS post explains how Amazon Bedrock prompt caching can reduce repeated input-token costs by up to 90 percent and lower time to first token when requests reuse the same context. It presents six scenarios using the Converse API, including document, system prompt, tool definition, mixed TTL, tenant-isolated, and LangChain caching. ### Source excerpt Prompt caching in Amazon Bedrock can cut input token costs by up to 90% when you repeatedly send the same context to foundation models. This post walks through six practical prompt caching scenarios using the Converse API: message content, system prompt, tool definition, mixed TTL, tenant isolation, and LangChain integration. ## Laravel MCP 1.0 Is Released DevFeed: [Laravel MCP 1.0 Is Released]() Original publisher: [Read original article]() Author: Eric L. Barnes Published: 2026-09-15T14:27:51Z Content type: release Language: en Sources: [Laravel]() Topics: [Laravel](), [Model Context Protocol (MCP)](), [OAuth](), [pkce](), [Caching](), [servers]() Tags: [caching](), [context-window](), [laravel](), [mcp](), [news](), [oauth](), [pkce](), [servers]() ### AI overview Laravel MCP 1.0 is the first stable release of Laravel's package for building Model Context Protocol servers. It adds protocol revision 2026-07-28 support, searchable tool catalogs, cache hints, stateless request handling, OAuth changes requiring PKCE, and Client ID Metadata Documents. ### Source excerpt Laravel MCP 1.0 ships protocol revision 2026-07-28, stateless servers, searchable tool catalogs, cache hints, and reworked OAuth with required PKCE. The post Laravel MCP 1.0 Is Released appeared first on Laravel News. Join the Laravel Newsletter to get Laravel articles like this directly in your inbox. ## Apollo Client 4.3 adds native custom scalar support and safer TypeScript types DevFeed: [Apollo Client 4.3 adds native custom scalar support and safer TypeScript types]() Original publisher: [Read original article]() Author: Jerel Miller Published: 2026-09-15T12:01:05Z Content type: release Language: en Sources: [Apollo Blog]() Topics: [GraphQL](), [TypeScript](), [Cache](), [Caching](), [JSON]() Tags: [cache](), [graphql](), [json](), [typescript](), [upgrade]() ### AI overview Apollo Client 4.3 adds native custom scalar integration, type-safe cache access, and improved TypeScript handling for incremental GraphQL data. ### Source excerpt Apollo Client 4.3 is here: native custom scalar support, type-safe cache access, and more consistent @defer and @stream behavior. See what's new and upgrade today. ## One or two nameservers? DevFeed: [One or two nameservers?]() Original publisher: [Read original article]() Author: Geoff Huston Published: 2026-09-15T06:01:49Z Content type: article Language: en Sources: [APNIC Blog]() Topics: [Internet](), [Query (disambiguation)](), [Server](), [Caching]() Tags: [2026](), [africa](), [article](), [china](), [dns](), [europe](), [experiment](), [india](), [internet](), [ipv4](), [ipv6](), [measurement](), [measurements](), [recursive-resolver](), [repeat](), [responses](), [server](), [tech-matters]() ### AI overview This article reports an experiment testing whether serving a DNS zone with two authoritative dual-stack nameservers changes repeated queries. Compared with one nameserver, two nameservers increased the single-query completion rate from 58% to 71% and reduced the average queries per test from 3.43 to 2.57. ### Source excerpt Do multiple dual-stack nameservers increase or decrease repeat DNS queries? The results of this experiment were a complete surprise. ## Agoda Replaces 72-Shard SQL Server Price Cache with DragonflyDB DevFeed: [Agoda Replaces 72-Shard SQL Server Price Cache with DragonflyDB]() Original publisher: [Read original article]() Author: Leela Kumili Published: 2026-09-14T13:48:00Z Content type: news Language: en Sources: [InfoQ]() Topics: [Cache](), [Caching](), [migration](), [sql-server](), [Latency](), [Prometheus]() Tags: [a-b-testing](), [agoda-price-cache-dragonflydb](), [architecture-design](), [cache](), [caching](), [development](), [devops](), [distributed-cache](), [distributed-data](), [distributed-systems](), [latency](), [microservices](), [migration](), [news](), [prometheus](), [redis](), [sql-server]() ### AI overview Agoda migrated its 1.5 TB hotel Price Cache from 72 Microsoft SQL Server shards to DragonflyDB. The staged migration used workload testing, dual reads, parity metrics, and gradual traffic shifting; Agoda reported substantially lower P99 read latency. ### Source excerpt Agoda migrated its 1.5 TB hotel Price Cache from 72 SQL Server shards to DragonflyDB to handle growing read and write volumes. The migration used staged dual reads, parity validation, gradual traffic shifting, and decentralized failover detection. Agoda reports an approximately eightfold reduction in P99 read latency, with two DragonflyDB clusters providing high availability. By Leela Kumili ## Using Exact-Match Response Caching to Reduce LLM Costs DevFeed: [Using Exact-Match Response Caching to Reduce LLM Costs]() Original publisher: [Read original article]() Author: Abhilash Rao Mesala Published: 2026-09-14T11:00:00Z Content type: article Language: en Sources: [The New Stack]() Topics: [Caching](), [Large Language Model]() Tags: [ai-engineering](), [ai-infrastructure](), [caching](), [contributed](), [cost](), [finops](), [generation](), [hash](), [llm](), [token]() ### AI overview The article explains how to reduce LLM costs by fingerprinting requests, context, model settings, and underlying data to create exact-match cache keys. Valid cached responses can be reused without calling the model. It distinguishes response caching from provider prompt caching, where only eligible prompt computation is reused. ### Source excerpt An LLM can answer the same question a thousand times and charge you each time. Before paying for another answer, The post Why an old caching trick is your secret to lower LLM costs appeared first on The New Stack. ## nginx: The Map Variable That Fixes the URI in proxy\_pass DevFeed: [nginx: The Map Variable That Fixes the URI in proxy\_pass]() Original publisher: [Read original article]() Author: Philipp Lehmann (philipp.lehmann@gruppe.ai) Published: 2026-09-14T07:00:00Z Content type: tutorial Language: en Sources: [Philipp Lehmann]() Topics: [nginx](), [proxy](), [Caching](), [configuration]() Tags: [cache-control](), [ci-cd](), [configuration](), [docker](), [networking](), [nginx](), [observability](), [proxy](), [reverse-proxy]() ### AI overview This article explains an nginx build failure caused by using a URI-bearing proxy_pass inside a regex location. A newer nginx version treated the previously accepted configuration as an error. The proposed fix uses a bare upstream in the regex location and a map on $uri to apply a path-specific Cache-Control policy. ### Source excerpt nginx: [emerg] invalid URI prefix in proxy_pass: a regex location needs a bare upstream, and a map on $uri keeps a path-specific Cache-Control header. ## Cloudflare Tests Cache Transcoding to Reduce Storage Requirements DevFeed: [Cloudflare Tests Cache Transcoding to Reduce Storage Requirements]() Original publisher: [Read original article]() Author: Renato Losio Published: 2026-09-13T10:35:00Z Content type: news Language: en Sources: [InfoQ]() Topics: [Caching](), [Transcodings](), [cloud-infrastructure](), [Pingora]() Tags: [architecture-design](), [cache](), [caching](), [cdn](), [cloud](), [cloudflare](), [cloudflare-cache-transcoding](), [compression](), [cpu](), [data-centers](), [development](), [edge-computing](), [news](), [pingora](), [rust](), [storage]() ### AI overview Cloudflare is testing Cache Transcoding, a prototype that Zstandard-compresses eligible uncompressed text before it is stored in cache. The approach aims to increase effective cache capacity and reduce inter-data-center transfer, with configurable CPU and storage trade-offs. ### Source excerpt Cloudflare recently described a prototype called Cache Transcoding that compresses eligible cache content, mainly uncompressed text such as HTML, JSON, CSS, and JavaScript, using Zstandard before storing it on disk. The hyperscaler estimates that the approach could provide petabytes of additional effective cache capacity, although broader testing is still needed. By Renato Losio ## Cache invalidation is a distributed systems problem DevFeed: [Cache invalidation is a distributed systems problem]() Original publisher: [Read original article]() Author: hello@ankit-rana.com Published: 2026-09-13T00:00:00Z Content type: tutorial Language: en Sources: [Ankit Rana | Mechanical Sympathy]() Topics: [Cache](), [Caching](), [distributed-systems]() Tags: [cache](), [cache-invalidation](), [caching](), [consistency](), [distributed](), [distributed-systems](), [ordering](), [race-conditions](), [redis](), [ttl]() ### AI overview Cache invalidation coordinates updates between a database and a cache without a shared transaction. The article explains how operation ordering can let a concurrent reader repopulate stale data and recommends writing to the database before invalidating the cache, followed by a second invalidation to close the remaining race. ### Source excerpt A cache and a database are two stores that must agree, which makes every invalidation a distributed transaction without a coordinator. The ordering matters more than the mechanism: invalidating before the database write leaves a window where a concurrent reader repopulates the cache with the old value and it stays wrong until the TTL expires. Deleting the key rather than writing the new value removes a whole class of ordering bug, because two concurrent deletes commute and two concurrent writes do not. ## Claims About RubyGems Caching and RubyDoc Code Execution DevFeed: [Claims About RubyGems Caching and RubyDoc Code Execution]() Original publisher: [Read original article]() Published: 2026-09-12T00:02:11Z Content type: opinion Language: en Sources: [Aaron Patterson]() Topics: [rubygems](), [Caching](), [vulnerability](), [Docker](), [Web Scraping]() Tags: [caching](), [docker](), [rubygems](), [scraping](), [vulnerability]() ### AI overview An opinion post discusses alleged RubyGems activity involving a caching vulnerability and web scraping. It also describes how published gems can cause RubyDoc.info to run YARD-loaded code in a networked Docker container. ### Source excerpt Today Reuters and the Wall Street Journal both reported about rogue AI agents at OpenAI attacking RubyGems.org. https://www.rubyhack.ai/ has an amazing writeup, and you should read it. I just wanted to make a quick post about it because it's wild. TL;DR: It seems like OpenAI Bots knew about the RubyGems caching vulnerability, tried to take advantage of it, and at the same time ran some weird web scraping code on RubyDoc.info. Back in May, socket.dev reported about a "GemStuffer Campaign" where someone (I guess OpenAI) was uploading tons of junk gems to RubyGems.org. For some reason, the gems would scrape UK government websites, then repackage the data as gems, and attempt to upload them to RubyGems. I honestly didn't think much about this (or even look into it) until Sydney Von Arx and Spencer Kitts (both co-authors on https://www.rubyhack.ai) contacted me asking about RubyGems. I thought the claims they were making were completely outlandish until I actually read the code in these "GemStuffer" gems. After reading the code in these gems, a couple things stood out to me. YARD Documentation First, the gems leverage YARD documentation to execute arbitrary code on host machines. In most of the examples you'll see a .yardopts file that looks like this: --load ./script.rb README.md lib/**/*.rb Here's a link to an example. If you have YARD installed, and you install this gem, then YARD will load and run whatever is in ./script.rb from inside the gem. I think it's pretty common knowledge that C extensions will execute extconf.rb (so you basically have an RCE vector), but I was surprised to find out that a documentation tool would do that too. Nobody is going to install a gem named slnleaker5 though, so why would this matter? Well, any time a Gem is published RubyDoc.info will download the gem and process the YARD documentation. RubyDoc.info will execute the arbitrary code inside a Docker container. The Docker container still has network access though, so these gems could ha ## How to call external APIs from Webflow without leaking API keys DevFeed: [How to call external APIs from Webflow without leaking API keys]() Original publisher: [Read original article]() Author: Ismail Ajagbe Published: 2026-09-12T00:00:00Z Content type: article Language: en Sources: [Webflow Blog]() Topics: [API](), [API keys](), [Authentication](), [Back end](), [Automation](), [Caching](), [Next.js](), [Node.js](), [GitHub]() Tags: [api-keys](), [apis](), [authentication](), [automation](), [backend](), [cache](), [github](), [guides](), [how-to](), [next-js](), [node-js]() ### AI overview A practical guide to integrating external APIs with Webflow through a Webflow Cloud server-side proxy, client-side fetch for public APIs, or no-code automation. It emphasizes keeping API credentials out of the browser and explains why the proxy is preferred for authenticated production integrations. ### Source excerpt Learn three ways to connect external APIs to Webflow (server-side proxy, client-side fetch, and no-code automation). ## DeepSeek's new model sets a template for powerful LLMs that run lean DevFeed: [DeepSeek's new model sets a template for powerful LLMs that run lean]() Original publisher: [Read original article]() Author: Tobias Mann Published: 2026-09-11T07:15:00Z Content type: news Language: en Sources: [www.theregister.com - Articles]() Topics: [deepseek](), [Large Language Model](), [Caching](), [Latency](), [Google]() Tags: [ai-and-ml](), [cache](), [compute](), [cost](), [datacenter](), [deepseek](), [embedding](), [flash](), [google](), [gpu](), [latency](), [llm](), [llms](), [model](), [release]() ### AI overview DeepSeek V4.1 Flash is a larger LLM whose architectural changes aim to reduce serving memory and compute needs. The article highlights lower KV-cache consumption, improved prompt processing, and an N-gram-based conditional memory module. ### Source excerpt DeepSeek V4.1 Flash proves that just because you build a bigger model doesn't mean you need more GPUs to serve it ## Choosing among write-through, write-behind, and write-around caching strategies DevFeed: [Choosing among write-through, write-behind, and write-around caching strategies]() Original publisher: [Read original article]() Author: hello@ankit-rana.com Published: 2026-09-11T00:00:00Z Content type: tutorial Language: en Sources: [Ankit Rana | Mechanical Sympathy]() Topics: [Caching](), [Databases](), [Redis](), [consistency](), [Latency]() Tags: [cache](), [caching](), [consistency](), [database](), [latency](), [orders](), [postgres](), [redis](), [strategies](), [system-design](), [write](), [write-behind](), [write-through]() ### AI overview This article compares write-through, write-behind, and write-around caching strategies. It explains their trade-offs among database latency, cache hit rates, consistency, and the risk of losing acknowledged writes when a process fails. ### Source excerpt The three write strategies differ in what happens between the cache write and the database write, and that gap is where correctness lives. Write-through updates both before acknowledging, so it is safe and pays the full database latency. Write-behind acknowledges after the cache write and flushes later, which is fast and can lose acknowledged writes if the process dies. Write-around skips the cache entirely on write, which avoids caching data nobody reads and costs a miss on the next read. ## Building Pinterest's VLM Serving Stack on NVIDIA Dynamo DevFeed: [Building Pinterest's VLM Serving Stack on NVIDIA Dynamo]() Original publisher: [Read original article]() Author: Pinterest Engineering Published: 2026-09-10T23:08:16Z Content type: article Language: en Sources: [Pinterest Engineering Blog - Medium]() Topics: [vlm](), [Inference](), [model-deployment](), [Orchestration](), [Latency](), [Caching](), [Open Source Models & Datasets]() Tags: [blackwell](), [cache](), [dynamo](), [embeddings](), [engineering](), [inference](), [latency](), [llm](), [multimodal](), [multimodal-ai](), [nvidia](), [nvidia-blackwell](), [open-source](), [orchestration](), [performance](), [pinterest](), [vllm](), [vlm](), [vlm-serving]() ### AI overview Pinterest describes its VLM serving stack built on NVIDIA Blackwell GPUs and NVIDIA Dynamo. The stack addresses multimodal inference demands such as image processing, variable prefill costs, KV-cache pressure, routing, and cache offloading. ### Source excerpt Lei Pan | Senior Software Engineer; Salina Wu | Senior Software Engineer; Cristian Lopez | Software Engineer I; Guangtong Bai | Staff Software Engineer; Soam Acharya | Principal Engineer; Saurabh Vishwas Joshi | Principal Engineer; Chia-Wei Chen | Staff Software Engineer; Ambud Sharma | Principal Engineer Why VLM Serving Matters at Pinterest Pinterest is a visual search and discovery platform, so its AI systems must reason over both language and visual content. Vision-language models (VLMs), which can interpret images, compare visual candidates, and respond naturally to user intent, are becoming the foundation for the next generation of Pinterest experiences: Pinterest Assistant, hybrid search, multimodal reranking, content understanding, signal generation, content safety, and more. This direction also reflects Pinterest's broader strategy to customize open-source models to meet its product & scale needs. Pinterest Assistant is a standout example. This multi-turn conversational experience covers both user language and visual content. Serving it requires low-latency VLM inference over rich multimodal context as well as reworking Qwen3-VL with proprietary multimodal embeddings to cut runtime cost while improving performance. Serving VLMs, however, introduces more challenges compared to text-only LLM workloads. Requests may carry multiple images, require extra vision encoder computation, incur larger and more variable prefill cost, and create higher KV cache pressure. To support this new class of models & product experiences, we built Pinterest's VLM serving stack on top of NVIDIA Blackwell GPUs and NVIDIA Dynamo. Blackwell GPUs incorporate many architectural innovations that are uniquely positioned for today's most demanding AI workloads -- including higher BF16/FP8 compute throughput, increased memory bandwidth, and larger HBM memory capacity -- that enable dramatically higher performance for inference. Dynamo provides a distributed inference orchestration layer that g ## Reduce LLM latency with prefix-aware routing on Amazon SageMaker Inference DevFeed: [Reduce LLM latency with prefix-aware routing on Amazon SageMaker Inference]() Original publisher: [Read original article]() Author: Kareem Syed-Mohammed Published: 2026-09-10T21:58:09Z Content type: release Language: en Sources: [Artificial Intelligence]() Topics: [Large Language Model](), [Low-Latency Inference](), [Caching](), [Inference](), [Inference Performance]() Tags: [advanced-300](), [amazon-sagemaker](), [amazon-sagemaker-ai](), [announcements](), [benchmarks](), [caching](), [inference](), [latency](), [llama](), [llm](), [routing](), [tensorrt-llm](), [vllm]() ### AI overview Amazon SageMaker Inference introduces prefix-aware routing for LLM requests. By consistently sending requests with matching prompt prefixes to the same instance, it improves reuse of cached KV computations and can reduce time to first token. ### Source excerpt Amazon SageMaker Inference now offers prefix-aware routing, a routing strategy that sends requests sharing the same prompt prefix to the same instance so the KV cache stays warm. In benchmarks on Llama 3.1 70B, it reduced P50 time-to-first-token by up to 77% and raised KV cache hit rates from about 25% to over 80%. [Next page]()