# Benchmarking tiered KV cache offloading on H100 GPUs in OpenShift AI 3.5

DevFeed: [Benchmarking tiered KV cache offloading on H100 GPUs in OpenShift AI 3.5](<https://devfeed.tech/articles/benchmarking-tiered-kv-cache-offloading-on-h100-gpus-in-openshift-ai-3-5-112341.md>)

Original publisher: [Read original article](<https://developers.redhat.com/articles/2026/11/02/benchmarking-tiered-kv-cache-offloading-h100-gpus-openshift-ai-3-5>)

Author: Ramesh Doddaiah, Alberto Perdomo, Nathan Scott, Ashish Kamra, Naina Singh

Published: 2026-11-02T14:17:22Z

Content type: news

Language: en

Sources: [Red Hat](<https://devfeed.tech/sources/red-hat.md>), [Red Hat Developer](<https://devfeed.tech/sources/red-hat-developer.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Cache](<https://devfeed.tech/topics/cache.md>), [context window](<https://devfeed.tech/topics/context-window.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-inference](<https://devfeed.tech/tags/ai-inference.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [cache](<https://devfeed.tech/tags/cache.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [offloading](<https://devfeed.tech/tags/offloading.md>), [openshift](<https://devfeed.tech/tags/openshift.md>), [openshift-ai](<https://devfeed.tech/tags/openshift-ai.md>)

## AI overview

In Red Hat OpenShift AI 3.5 tests on H100 GPUs, tiered KV cache offloading improved long-context inference most when the reusable working set exceeded CPU cache capacity. For Nemotron-Ultra-253B, adding NVMe or CephFS to CPU offload raised throughput from 0.44 to about 0.76-0.77 requests/s and cut mean TTFT from about 60 seconds without offload to 9.6 seconds; CPU alone reached 0.44 requests/s. Qwen's working set fit in CPU memory, so CPU offload captured most gains, while secondary storage added about 10% in a single-replica test. Gemma's gains were limited by GPU memory utilization. Local NVMe had the lowest storage latency and suits single-node locality; CephFS adds latency but offers shared capacity and cross-replica cache reuse, especially as concurrency rises. The results used CephFS with a single data copy, so affected entries would need recomputation after OSD loss. The article recommends measuring block-sized storage p99 latency, estimating the restore-versus-recompute crossover, then validating candidate tiers against representative traces and concurrency.

## Source excerpt

Long-context workloads are quickly becoming the norm for enterprise AI. Agentic applications carry an ever-growing history of tool calls and intermediate results, retrieval-augmented generation prepends thousands of tokens of retrieved context, and multi-turn assistants resend the whole conversation with every turn. In all of these cases, the model sees large parts of the same prompt again and again. The post Benchmarking tiered KV cache offloading on H100 GPUs in OpenShift AI 3.5 appeared first on Red Hat Developer.