# StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference

DevFeed: [StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference](<https://devfeed.tech/articles/strataflux-memory-hierarchy-native-low-bit-llm-inference-120736.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.08363>)

Author: Jake Hyun, Ziyu Zhao, Jae-sun Seo, Mohamed S. Abdelfattah

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Memory Heirarchy](<https://devfeed.tech/topics/memory-heirarchy.md>), [vllm](<https://devfeed.tech/topics/vllm.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [2025](<https://devfeed.tech/tags/2025.md>), [3](<https://devfeed.tech/tags/3.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [aime](<https://devfeed.tech/tags/aime.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [awq](<https://devfeed.tech/tags/awq.md>), [bandwidth](<https://devfeed.tech/tags/bandwidth.md>), [base](<https://devfeed.tech/tags/base.md>), [but](<https://devfeed.tech/tags/but.md>), [cache](<https://devfeed.tech/tags/cache.md>), [can](<https://devfeed.tech/tags/can.md>), [challenges](<https://devfeed.tech/tags/challenges.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [hierarchy](<https://devfeed.tech/tags/hierarchy.md>), [inference](<https://devfeed.tech/tags/inference.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [memory](<https://devfeed.tech/tags/memory.md>), [memory-hierarchy](<https://devfeed.tech/tags/memory-hierarchy.md>), [native](<https://devfeed.tech/tags/native.md>)

## AI overview

StrataFlux uses host RAM and SSDs to restore some accuracy lost to low-bit weight and KV-cache quantization while retaining low GPU memory use. It selects and streams extra precision, overlaps transfers with computation, and reuses fetched data across decoding steps to limit waiting. In the reported results, 2-bit Llama-3.1-8B-Instruct weights improve WikiText-2 perplexity from 9.46 to 8.40 while retaining 95.4% of quantized-base throughput on RTX 5090. For Qwen3-8B with 2-bit weights and KV, AIME 2024/2025 accuracy rises from 11.0% to 47.1% at unchanged GPU memory use. At 128K context, 3-bit weights and 2-bit KV achieve comparable perplexity and speed to vLLM's 4-bit AWQ weights and 8-bit KV, using about half the GPU memory. The supplied text is an abstract, so it does not provide implementation details or broader evaluation results.

## Source excerpt

arXiv:2611.08363v1 Announce Type: new Abstract: Quantization reduces the memory and bandwidth demands of large language model (LLM) inference but can degrade accuracy; offloading higher-precision data can instead stall generation. We introduce StrataFlux, a framework that challenges this tradeoff by recovering accuracy lost to low-bit weight and key-value (KV) cache quantization while retaining quantization's memory and speed advantages. StrataFlux co-designs refinement representation, selection, and delivery to make host RAM and solid-state drives useful sources of additional precision. Its selective precision streaming reduces transfer volume, overlaps delivery with computation, and reuses fetched precision across decoding steps so generation need not wait for every transfer. We evaluate 2-4-bit weights and 2-3-bit KV-cache on dedicated GPUs and unified-memory platforms. For 2-bit Llama-3.1-8B-Instruct weights, StrataFlux reduces WikiText-2 perplexity from 9.46 to 8.40 while maintaining 95.4% of the quantized base's throughput on RTX 5090. With 2-bit weights and KV, StrataFlux raises Qwen3-8B's AIME 2024/2025 accuracy from 11.0% to 47.1% at unchanged GPU memory use. At 128K context, Llama-3.1-8B-Instruct with 3-bit weights and 2-bit KV achieves comparable perplexity and speed to vLLM's 4-bit AWQ weights and 8-bit KV on RTX 5090, using approximately half the GPU memory. These results show that GPU-resident precision need not set the accuracy ceiling, nor offloading-tier latency dictate decoding speed.