# quantization

Quantization is a technique for representing numerical values, model weights, and activations with lower-precision data types to reduce memory usage and inference costs in neural-network computation.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Running an Embedding Model Next to My Notes on an iPhone

DevFeed: [Running an Embedding Model Next to My Notes on an iPhone](<https://devfeed.tech/articles/running-an-embedding-model-next-to-my-notes-on-an-iphone-146222.md>)

Original publisher: [Read original article](<https://hackernoon.com/running-an-embedding-model-next-to-my-notes-on-an-iphone?source=rss>)

Author: Lavneet Sharma

Published: 2026-11-20T09:30:03Z

Content type: article

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [llama.cpp](<https://devfeed.tech/topics/llama-cpp.md>), [iphone](<https://devfeed.tech/topics/iphone.md>), [quantization](<https://devfeed.tech/topics/quantization.md>)

Tags: [app](<https://devfeed.tech/tags/app.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cpp](<https://devfeed.tech/tags/cpp.md>), [device](<https://devfeed.tech/tags/device.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [embedding-model](<https://devfeed.tech/tags/embedding-model.md>), [embedding-models](<https://devfeed.tech/tags/embedding-models.md>), [executorch](<https://devfeed.tech/tags/executorch.md>), [iphone](<https://devfeed.tech/tags/iphone.md>), [llama](<https://devfeed.tech/tags/llama.md>), [llama-cpp](<https://devfeed.tech/tags/llama-cpp.md>), [local-ai-models](<https://devfeed.tech/tags/local-ai-models.md>), [model](<https://devfeed.tech/tags/model.md>), [note-taking-app](<https://devfeed.tech/tags/note-taking-app.md>), [notes](<https://devfeed.tech/tags/notes.md>), [on-device](<https://devfeed.tech/tags/on-device.md>), [on-device-ai](<https://devfeed.tech/tags/on-device-ai.md>), [parameter](<https://devfeed.tech/tags/parameter.md>), [private](<https://devfeed.tech/tags/private.md>), [react-native](<https://devfeed.tech/tags/react-native.md>), [running](<https://devfeed.tech/tags/running.md>), [search](<https://devfeed.tech/tags/search.md>), [semantic](<https://devfeed.tech/tags/semantic.md>), [semantic-search](<https://devfeed.tech/tags/semantic-search.md>), [sqlite](<https://devfeed.tech/tags/sqlite.md>), [sqlite-fts5](<https://devfeed.tech/tags/sqlite-fts5.md>)

### AI overview

VB Notes combines on-device semantic search with keyword search while keeping notes as plain Markdown files. On iPhone, LFM2.5-Embedding-350M was the default after ranking the intended note first in 17 of 18 author-written paraphrase tests; smaller all-MiniLM-L6-v2 scored 16 of 18, while the multilingual MiniLM matched across Hindi and English in 19 of 22 tests but produced more unrelated matches. The app chunks notes with titles, stores scaled int8 vectors in a cache SQLite database, and filters results with a model-specific score floor plus a window below the best match. It merges semantic and SQLite FTS5 keyword results by reciprocal rank fusion. Desktop uses the same model and chunking through llama.cpp, with Metal on Apple Silicon and CPU elsewhere. The findings are based on a small, author-designed test set.

### Source excerpt

How I built private, on-device semantic search for a notes app using a 350M-parameter embedding model, SQLite, ExecuTorch, and llama.cpp.

## Achieving Near-Zero-Overhead Multi-Model Hierarchical Classification in Real-Time Detection Pipelines

DevFeed: [Achieving Near-Zero-Overhead Multi-Model Hierarchical Classification in Real-Time Detection Pipelines](<https://devfeed.tech/articles/achieving-near-zero-overhead-multi-model-hierarchical-classification-in-real-time-detection-pipelines-144919.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2608.11770>)

Author: Vaishnav Raju

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [Jetson Orin](<https://devfeed.tech/topics/jetson-orin.md>), [AI Integration Strategies](<https://devfeed.tech/topics/ai-integration-strategies.md>), [real-time](<https://devfeed.tech/topics/real-time.md>)

Tags: [accelerators](<https://devfeed.tech/tags/accelerators.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [autonomous](<https://devfeed.tech/tags/autonomous.md>), [autonomous-vehicles](<https://devfeed.tech/tags/autonomous-vehicles.md>), [bottleneck](<https://devfeed.tech/tags/bottleneck.md>), [classification](<https://devfeed.tech/tags/classification.md>), [classifier](<https://devfeed.tech/tags/classifier.md>), [classifiers](<https://devfeed.tech/tags/classifiers.md>), [compilation](<https://devfeed.tech/tags/compilation.md>), [concurrent](<https://devfeed.tech/tags/concurrent.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [custom](<https://devfeed.tech/tags/custom.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [detection](<https://devfeed.tech/tags/detection.md>), [drone](<https://devfeed.tech/tags/drone.md>), [edge](<https://devfeed.tech/tags/edge.md>), [overhead](<https://devfeed.tech/tags/overhead.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [real-time](<https://devfeed.tech/tags/real-time.md>)

### AI overview

The paper reports that a GPU detector paired with DLA classification on a Jetson Orin NX ran at 12.5 FPS at 1080p, compared with 13.3 FPS for the detector alone. Its five-step deployment method covers adapting classification architectures, repairing TensorRT implicit quantization ranges for pipeline validation (raising reported accuracy from 75% to 94%), quantization-aware training, ONNX graph surgery, and concurrent GPU/DLA inference. The authors also describe nine engineering constraints and report dual-DLA scaling without additional cost. Evidence here is limited to the abstract.

### Source excerpt

arXiv:2608.11770v2 Announce Type: replace-cross Abstract: Edge-deployed vision systems in target recognition, surveillance, autonomous vehicles, and drone domains require hierarchical inference pipelines where a detection model identifies objects of interest and downstream classifiers provide fine-grained attribute analysis. Running all models on the GPU creates a serial bottleneck that limits real-time throughput as pipeline stages grow. Modern edge SoCs pair GPUs with dedicated neural accelerators (NPUs, DLAs) capable of concurrent execution, yet deploying custom models on these accelerators remains impractical due to strict operator constraints, quantization incompatibilities, and an undocumented end-to-end pipeline. We target NVIDIA Jetson DLA cores as the representative platform. We present a five-step methodology for zero GPU fallback DLA INT8 deployment of classification backbones, comprising architecture adaptation, manual dynamic range workaround to rescue TensorRT's implicit quantization (recovering 94.0% accuracy from implicit quantization's 75%) for rapid pipeline validation before explicit quantization, quantization-aware training, ONNX graph surgery for DLA compilation, and a concurrent GPU-detection/DLA-classification inference pipeline. We document nine engineering constraints with root-cause analysis and generalizable solutions. Validation on a dual-head person attribute classifier running on DLA alongside a GPU object detector on a Jetson Orin NX demonstrates near-zero pipeline overhead (12.5 vs. 13.3~FPS detector-only at 1080p), with dual-DLA scaling at no additional cost. The methodology is backbone-agnostic and generalizes to any detection-classification edge pipeline.

## Tetra: Serving Leech-Lattice Quantized LLMs at 2.7 Bits per Parameter

DevFeed: [Tetra: Serving Leech-Lattice Quantized LLMs at 2.7 Bits per Parameter](<https://devfeed.tech/articles/tetra-serving-leech-lattice-quantized-llms-at-2-7-bits-per-parameter-144940.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.35465>)

Author: Pier-Jean Malandrino

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>)

Tags: [awq](<https://devfeed.tech/tags/awq.md>), [block](<https://devfeed.tech/tags/block.md>), [code](<https://devfeed.tech/tags/code.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [engine](<https://devfeed.tech/tags/engine.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [figure](<https://devfeed.tech/tags/figure.md>), [file](<https://devfeed.tech/tags/file.md>), [files](<https://devfeed.tech/tags/files.md>), [fp16](<https://devfeed.tech/tags/fp16.md>), [full](<https://devfeed.tech/tags/full.md>), [generate](<https://devfeed.tech/tags/generate.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpu-memory](<https://devfeed.tech/tags/gpu-memory.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [lattice](<https://devfeed.tech/tags/lattice.md>), [llms](<https://devfeed.tech/tags/llms.md>), [lookup](<https://devfeed.tech/tags/lookup.md>), [serving](<https://devfeed.tech/tags/serving.md>), [tetra](<https://devfeed.tech/tags/tetra.md>)

### AI overview

Tetra makes Leech-lattice quantization practical for GPU inference by encoding each 24-weight block in 48 bits and decoding it during matrix-vector multiplication with six table loads and two small lookups. Its shared 16 KiB table avoids the enormous full codebook, and the kernel reads 2.148 bits per weight, compared with 4.804 bits per weight for the authors' earlier load-time expansion approach. Full Qwen3-4B, 8B and 14B models occupy 2.70-2.73 bits per parameter, using row-specific scales, 4-bit storage for the matrices that lose most quality, and 4-bit embedding tables. On MMLU they trail 4-bit AWQ models (5.3-6.0 bits per parameter) by 2.46-4.76 points; on GSM8K they trail FP16 by 3.26-9.63 points. Reported generation rates are 57.2-113.8 tokens per second. At 4B, Tetra scores 23.6 points above llama.cpp IQ2_XXS. All reported measurements used one NVIDIA L40S GPU.

### Source excerpt

arXiv:2609.35465v2 Announce Type: replace Abstract: Leech-lattice quantization gives good quality at two bits per weight, but its codebooks hold more than 10^14 points, too many for a lookup table. Our earlier kernel expanded the codes at load time and read 4.804 bits per weight from GPU memory for 2 bits of code. We present Tetra, a new codebook on the same lattice. A 24-weight block still takes 48 bits, most of which index a 64-state trellis of the Golay code and one shared 16 KiB table. The kernel decodes a block with six table loads and two small lookups inside the matrix-vector product, and reads 2.148 bits per weight. For full models, we retrain one scale per matrix row, store the matrices that lose the most as 4-bit integers, and pay for them with 4-bit embedding tables. Our Qwen3-4B, 8B and 14B files hold 2.73, 2.70 and 2.73 bits per parameter over the whole model. They score 63.37, 69.58 and 75.66 on the full MMLU test set, 4.76, 4.21 and 2.46 points below 4-bit AWQ at 5.3 to 6.0 bits per parameter. They generate 113.8, 95.0 and 57.2 tokens per second in our engine. On GSM8K, through the served kernel, they lose 9.63, 4.62 and 3.26 points to FP16. At 4B our file scores 23.6 points above llama.cpp's IQ2_XXS (2.48 bits per parameter). Every number we measured for a table or figure comes from one NVIDIA L40S GPU. We preregistered the main experiments.

## Online Temporal Rotations for 2 Bit KV-Cache Quantization in Video Diffusion Transformers

DevFeed: [Online Temporal Rotations for 2 Bit KV-Cache Quantization in Video Diffusion Transformers](<https://devfeed.tech/articles/online-temporal-rotations-for-2-bit-kv-cache-quantization-in-video-diffusion-transformers-145656.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26334>)

Author: Ryan Lucas, Petros Prastakos, Rahul Mazumder

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [diffusion-transformers](<https://devfeed.tech/topics/diffusion-transformers.md>)

Tags: [cache](<https://devfeed.tech/tags/cache.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [diffusion-transformers](<https://devfeed.tech/tags/diffusion-transformers.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [online](<https://devfeed.tech/tags/online.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [rotation](<https://devfeed.tech/tags/rotation.md>), [temporal](<https://devfeed.tech/tags/temporal.md>), [triton](<https://devfeed.tech/tags/triton.md>), [video](<https://devfeed.tech/tags/video.md>), [video-generation](<https://devfeed.tech/tags/video-generation.md>)

### AI overview

Temporal Rotation Quantization (TRQ) reduces quantization error in video diffusion KV caches by rotating temporally adjacent cache frames before 2-bit quantization. For each block, it fits an orthogonal transform to flatten per-channel means, using a tree of two-bit planar rotations followed by a fixed Hadamard transform. The rotations are fitted online for each video without gradients or model-weight changes; the authors say a greedy fit is globally optimal for this objective and provide fused Triton inference kernels. At matched compression, TRQ raises VBench subject consistency over QVG from 90.38 to 92.02 on LongCat-Video-13B and from 86.33 to 87.61 on HY-WorldPlay-8B; the abstract also reports gains on other metrics without specifying them.

### Source excerpt

arXiv:2611.26334v1 Announce Type: cross Abstract: Video diffusion transformers reuse key--value (KV) caches across denoising steps, which quickly becomes a bottleneck for long video generation. We find that quantizing temporally distant cache vectors leads to a large range and high quantization error, but that nearby video tokens in the KV-cache share a persistent pattern per channel. These patterns also vary across videos, motivating rotations that are adapted to each video's unique cache. We propose Temporal Rotation Quantization (TRQ), a method that rotates frames in the cache that are adjacent in time before quantization. For each block of temporally adjacent cache vectors, TRQ fits an orthogonal transform that maps the block's per-channel means to a vector that is close to a constant across channels. To keep the quantization format lightweight, we parameterize this rotation as a tree of $2 \times 2$ planar rotations, each stored with only two bits, followed by a fixed Hadamard transform shared across all blocks. The rotations are fit without gradients or changes to model weights, and we prove that a simple greedy fit is globally optimal for our 'mean-flattening' objective. The transforms are fit online for each new video, and we release fused Triton kernels to make this process efficient at inference. At 2-bit precision and matched compression, TRQ improves VBench subject consistency over the state-of-the-art method QVG from 90.38 to 92.02 on LongCat-Video-13B and from 86.33 to 87.61 on HY-WorldPlay-8B, as well as improving other metrics.

## Beam Model Technical Report

DevFeed: [Beam Model Technical Report](<https://devfeed.tech/articles/beam-model-technical-report-145663.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26358>)

Author: Reflection Team, Adam Abdulhamid, Osama Abuelsorour, Yasmin Alameddine, Carlo Alfano, Duarte Alves, Brandon Amos, Selvaraj Anandaraj, Arseni Anisimovich, George Antoniadis, Ioannis Antonoglou, Sheila

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [context window](<https://devfeed.tech/topics/context-window.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [Model Optimizer](<https://devfeed.tech/topics/model-optimizer.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>)

Tags: [agentic-workloads](<https://devfeed.tech/tags/agentic-workloads.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [beam](<https://devfeed.tech/tags/beam.md>), [co-design](<https://devfeed.tech/tags/co-design.md>), [coding](<https://devfeed.tech/tags/coding.md>), [context-window](<https://devfeed.tech/tags/context-window.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [fp8](<https://devfeed.tech/tags/fp8.md>), [gb300](<https://devfeed.tech/tags/gb300.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [nvfp4](<https://devfeed.tech/tags/nvfp4.md>), [open](<https://devfeed.tech/tags/open.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [speculative-decoding](<https://devfeed.tech/tags/speculative-decoding.md>), [throughput](<https://devfeed.tech/tags/throughput.md>)

### AI overview

Beam is a 501.3B-parameter sparse MoE model with 23.4B active parameters, designed for coding, reasoning, and agentic workloads. The abstract reports a 1M-token context window enabled by two-stage midtraining on verifiable, correctness-anchored data, following pretraining on 23.8T tokens. Post-training used asynchronous RL on 10.5K GB300 GPUs, producing over 100M rollouts in four weeks; measured capabilities continued to improve as RL compute increased, with no plateau reported. Hardware-model co-design, FP8/NVFP4 quantization, and a 1.93B-parameter DFlash 2 speculative decoding drafter target serving throughput. The report claims 3-4x less inference compute than many comparable models. On its safety evaluation, Beam refused or pivoted on 88.9% of harmful requests while complying with 81.5% of harmless requests; the text says its harmful-request refusal/pivot rate was highest among open-weight models on the Pareto frontier.

### Source excerpt

arXiv:2611.26358v1 Announce Type: cross Abstract: Beam is a sparse Mixture-of-Experts (MoE) model with 501.3B total parameters and 23.4B active parameters, specifically built and optimized for coding, reasoning, and agentic workloads. Beam delivers frontier level reasoning, while requiring 3-4x less inference compute than many models of its class. Pretrained on 23.8T diverse, curated, high-quality tokens, Beam features a 1M-token context window achieved through a novel two-stage midtraining phase, built on verifiable, correctness-anchored data. Post-training leveraged a large-scale, fully asynchronous reinforcement learning (RL) campaign across 10.5K GB300 GPUs, generating over 100M rollouts over 4 weeks of training. We believe this is one of the largest scale RL runs conducted by any open lab to date. Across our evaluation suite, capabilities continued to improve as we increased RL compute, with no sign of a plateau. Supported by deep hardware-model co-design, FP8/NVFP4 quantization, and a 1.93B-parameter speculative decoding drafter (DFlash 2), Beam maximizes serving throughput with efficient, flexible deployment. Averaged across conversation turns and harm categories, Beam refuses or pivots on 88.9% of harmful requests while complying with 81.5% of harmless requests, achieving the highest refusal/pivot rate on harmful requests among open-weight models on the Pareto frontier.

## Chameleon: Dynamic Format Adapter for Efficient Diffusion

DevFeed: [Chameleon: Dynamic Format Adapter for Efficient Diffusion](<https://devfeed.tech/articles/chameleon-dynamic-format-adapter-for-efficient-diffusion-144937.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.33496>)

Author: Arnab Sanyal, Sandeep Chinchali

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [diffusion-transformers](<https://devfeed.tech/topics/diffusion-transformers.md>)

Tags: [activation](<https://devfeed.tech/tags/activation.md>), [adapter](<https://devfeed.tech/tags/adapter.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [diffusion-models](<https://devfeed.tech/tags/diffusion-models.md>), [diffusion-transformers](<https://devfeed.tech/tags/diffusion-transformers.md>), [format](<https://devfeed.tech/tags/format.md>), [framework](<https://devfeed.tech/tags/framework.md>), [int8](<https://devfeed.tech/tags/int8.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [kurtosis](<https://devfeed.tech/tags/kurtosis.md>), [layer](<https://devfeed.tech/tags/layer.md>), [layers](<https://devfeed.tech/tags/layers.md>), [lookup](<https://devfeed.tech/tags/lookup.md>), [memory](<https://devfeed.tech/tags/memory.md>), [snr](<https://devfeed.tech/tags/snr.md>), [statistics](<https://devfeed.tech/tags/statistics.md>), [table](<https://devfeed.tech/tags/table.md>), [time](<https://devfeed.tech/tags/time.md>), [transformers](<https://devfeed.tech/tags/transformers.md>), [weight](<https://devfeed.tech/tags/weight.md>)

### AI overview

Chameleon improves diffusion-model post-training quantization by selecting the number format at a fixed bit-width to fit each weight channel and activation layer, with optional refinement by timestep. It chooses activation formats from kurtosis and diffusion SNR statistics, and weight formats offline by reconstruction error. The abstract reports that per-layer selection provides most of the gain, while timestep refinement helps mainly with 6-bit activations. Across SDXL, SDXL-Turbo and PixArt-α on COCO-2014, Chameleon achieved the best FID in six backbone and bit-width settings; CLIP scores were within 0.24 of the FP16 reference. On PixArt-α, a mixed-format kernel ran 1.09x faster than FP16 end-to-end with weights three times smaller. These are results reported in the supplied abstract excerpt.

### Source excerpt

arXiv:2609.33496v3 Announce Type: replace Abstract: Post-training quantization (PTQ) is the standard way to run modern diffusion models on memory-constrained accelerators, yet every existing diffusion PTQ scheme fixes the $\mathit{number\ format}$ in advance and only tunes the scale, zero point, or per-layer bit-width. At a fixed bit-width the best format depends on the distribution being encoded, and that distribution differs across weight channels, across layers, and along the diffusion timestep, where activation distributions slide from heavy-tailed and noise-dominated to tightly clustered and structured. We propose Chameleon, a PTQ framework that holds the bit-width fixed and treats the format itself as a discrete variable, chosen per weight channel and per activation layer, optionally refined per timestep bucket. Activation formats come from {INT8, FP8 E4M3, FP8 E5M2, MXFP8, MXINT8}, selected ahead of time from two cheap statistics (empirical kurtosis and the closed-form diffusion SNR) and stored in a lookup table; weight formats come from {INT8, MXINT8} at 8 bits or {INT4, NF4, FP4 E2M1, MXINT4, MXFP4} at 4 bits, selected offline by reconstruction error. An architectural fork adapts the same selection layer to multi-step UNets, single-step distilled models, and Diffusion Transformers. Most of the gain comes from the per-layer choice; the timestep refinement helps mainly at 6-bit activations. Across SDXL, SDXL-Turbo, and PixArt-$\alpha$ on COCO-2014, Chameleon achieves the best FID in all six backbone $\times$ bit-width settings, with CLIP within 0.24 of the FP16 reference and the best of all quantized methods at $W_{4}A_{8}$. On PixArt-$\alpha$, a real mixed-format kernel runs 1.09$\times$ faster than FP16 end-to-end with 3$\times$ smaller weights.

## TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks

DevFeed: [TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks](<https://devfeed.tech/articles/triaxialkv-toward-extreme-low-precision-kv-cache-quantization-for-agentic-inference-tasks-144895.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.17170>)

Author: Hanzhang Shen, Haoran Wu, Yiren Zhao, Robert Mullins

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Cache](<https://devfeed.tech/topics/cache.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-workloads](<https://devfeed.tech/tags/agentic-workloads.md>), [bf16](<https://devfeed.tech/tags/bf16.md>), [budget](<https://devfeed.tech/tags/budget.md>), [cache](<https://devfeed.tech/tags/cache.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [chat](<https://devfeed.tech/tags/chat.md>), [compression](<https://devfeed.tech/tags/compression.md>), [computer](<https://devfeed.tech/tags/computer.md>), [computer-use](<https://devfeed.tech/tags/computer-use.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [custom](<https://devfeed.tech/tags/custom.md>), [decode](<https://devfeed.tech/tags/decode.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [image](<https://devfeed.tech/tags/image.md>), [implement](<https://devfeed.tech/tags/implement.md>), [inference](<https://devfeed.tech/tags/inference.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [precision](<https://devfeed.tech/tags/precision.md>), [quantization](<https://devfeed.tech/tags/quantization.md>)

### AI overview

TriAxialKV reduces KV-cache memory for agentic inference by tagging tokens across recency, modality, and semantic role, calibrating sensitivity for each tag, and assigning INT2 or INT4 precision within a fixed memory budget. Its serving system combines calibration, quantization and memory management, and fused Triton decode kernels. In the reported OSWorld computer-use evaluation with Qwen3-VL-32B-Thinking, it matched SGLang's accuracy using a BF16 cache while supporting 4.5x KV-cache capacity and 30% higher end-to-end throughput on real GPUs.

### Source excerpt

arXiv:2605.17170v2 Announce Type: replace Abstract: Agentic workloads have emerged as a major workload for LLM inference. They differ significantly from chat-only workloads, requiring long-context processing, the ability to handle multimodal inputs, and structured multi-turn interactions with tool calling capabilities. As a result, their context exhibits structure that can carry different importance along three key axes: temporal recency to the current turn, modality such as text or image tokens, and semantic role such as user queries, tool calls, observations, or reasoning. These axes capture distinct token behaviors and lead to different sensitivities to KV-cache compression. However, existing KV-cache quantization methods are typically homogeneous or exploit only heterogeneity on a single dimension, such as temporal proximity or modality, overlooking the interactions among them. To this end, we introduce TriAxialKV, a novel mixed-precision KV-cache quantization scheme that assigns each token a triaxial tag, calibrates per-tag sensitivity, and allocates INT2/INT4 bit-widths under a fixed memory budget. We implement TriAxialKV as an end-to-end serving system, comprising calibration, mixed-precision quantization and memory management, and custom fused Triton decode kernels. When using Qwen3-VL-32B-Thinking as a computer-use agent operating the OSWorld, TriAxialKV matches the accuracy of SGLang with BF16 KV cache while supporting 4.5$\times$ KV cache capacity and achieving 30% higher end-to-end throughput, when running on real GPU systems.

## Scaling classical hardware for accurate real-time quantum error correction

DevFeed: [Scaling classical hardware for accurate real-time quantum error correction](<https://devfeed.tech/articles/scaling-classical-hardware-for-accurate-real-time-quantum-error-correction-145183.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26447>)

Author: Chenyang Jin, Susanne F. Yelin, Andi Gu

Published: 2026-11-20T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Quantum Computing](<https://devfeed.tech/topics/quantum-computing.md>), [fpga](<https://devfeed.tech/topics/fpga.md>), [quantization](<https://devfeed.tech/topics/quantization.md>)

Tags: [arithmetic](<https://devfeed.tech/tags/arithmetic.md>), [bicycle](<https://devfeed.tech/tags/bicycle.md>), [color](<https://devfeed.tech/tags/color.md>), [cs-ar](<https://devfeed.tech/tags/cs-ar.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [custom-silicon](<https://devfeed.tech/tags/custom-silicon.md>), [decoder](<https://devfeed.tech/tags/decoder.md>), [fpga](<https://devfeed.tech/tags/fpga.md>), [fpgas](<https://devfeed.tech/tags/fpgas.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [latency](<https://devfeed.tech/tags/latency.md>), [neural](<https://devfeed.tech/tags/neural.md>), [quant-ph](<https://devfeed.tech/tags/quant-ph.md>), [quantum-error-correction](<https://devfeed.tech/tags/quantum-error-correction.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [relay](<https://devfeed.tech/tags/relay.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [surface](<https://devfeed.tech/tags/surface.md>)

### AI overview

Rapids is a streaming neural decoder co-designed with FPGA hardware, using operations that map cheaply to the device and training in its integer arithmetic. On a single FPGA, it decodes each distance-19 surface-code syndrome round in under a microsecond; the abstract reports a logical error rate 240 times lower than existing real-time hardware decoders. The design also covers the color code and [[144,12,12]] bivariate-bicycle code, where it is reported to be 16 times more accurate than the real-time Relay decoder. Spreading Rapids across multiple FPGAs or implementing it in custom silicon can, according to the abstract, increase speed or accuracy or support larger codes.

### Source excerpt

arXiv:2611.26447v1 Announce Type: cross Abstract: Fault-tolerant quantum computation requires a classical decoder fast enough to process syndrome measurements at the rate at which they are generated, and to return its decisions with low latency. This is most demanding for superconducting processors, whose fast syndrome cycles force a trade-off between decoding speed and accuracy. Decoders fast enough to keep up sacrifice accuracy, which must be compensated for with larger codes and thus additional physical qubits. Here we introduce Rapids, a streaming neural decoder that is both fast and accurate. We achieve this by co-designing the decoder with a field-programmable gate array (FPGA): the network is built from operations that map cheaply onto the device and trained in its integer arithmetic. For the distance-19 surface code, Rapids decodes each round of syndromes in under a microsecond on a single FPGA, fast enough for superconducting processors, and returns its decisions with the low latency that fault-tolerant algorithms require. Its logical error rate is 240 times lower than that of existing real-time hardware decoders. The same design extends to the color code and the [[144,12,12]] bivariate-bicycle code, on which our decoder is 16 times more accurate than the real-time Relay decoder. We further show that the decoder scales: spread over several FPGAs or hardened into custom silicon, it becomes faster, more accurate, or able to decode larger codes. Classical hardware thus becomes a further resource, alongside qubits, that can be scaled to improve fault-tolerant quantum computers.

## NariQ: Sub-3-Bit Quantization for Long-Horizon Agentic Tasks

DevFeed: [NariQ: Sub-3-Bit Quantization for Long-Horizon Agentic Tasks](<https://devfeed.tech/articles/nariq-sub-3-bit-quantization-for-long-horizon-agentic-tasks-145146.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26157>)

Author: Doyeob Kim

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [Post-training optimization](<https://devfeed.tech/topics/post-training-optimization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [AI Agent](<https://devfeed.tech/topics/ai-agent.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-benchmark](<https://devfeed.tech/tags/agentic-benchmark.md>), [agentic-tasks](<https://devfeed.tech/tags/agentic-tasks.md>), [aime](<https://devfeed.tech/tags/aime.md>), [bf16](<https://devfeed.tech/tags/bf16.md>), [bits](<https://devfeed.tech/tags/bits.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [deepswe](<https://devfeed.tech/tags/deepswe.md>), [gguf](<https://devfeed.tech/tags/gguf.md>), [inference](<https://devfeed.tech/tags/inference.md>), [post-training](<https://devfeed.tech/tags/post-training.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [qwen3](<https://devfeed.tech/tags/qwen3.md>), [qwen3-8-27b](<https://devfeed.tech/tags/qwen3-8-27b.md>), [terminal-bench](<https://devfeed.tech/tags/terminal-bench.md>), [trellis](<https://devfeed.tech/tags/trellis.md>)

### AI overview

NariQ targets a gap in low-bit quantization for long-horizon agent tasks: methods calibrated on short web text can retain performance on AIME while doing substantially worse on DeepSWE. Its trellis-coded post-training method calibrates on tokens generated by the BF16 model during agent sessions. For Qwen3.8-27B, the 3.0- and 2.5-bit-per-weight versions are reported to match BF16 on DeepSWE and Terminal Bench, as well as AIME and RealWorldQA; the 2.0-bit checkpoint trails that performance. The authors describe these as the first quantizations at or below 3 bits shown to match full precision on long-horizon agent tasks. They provide GGUF files and inference through a llama.cpp fork.

### Source excerpt

arXiv:2611.26157v1 Announce Type: new Abstract: Long-horizon agentic tasks require a model to call tools over many steps at contexts of 100K tokens or more, yet many low-bit quantization methods are calibrated on short web text and judged on perplexity. This mismatch is costly: existing quantizations of Qwen3.8-27B at or below 3 bits per weight match BF16 on shorter tasks like AIME, yet in published results solve at most 72% as many tasks as BF16 on DeepSWE, a long-horizon agentic benchmark. We propose NariQ, a trellis-coded post-training quantization method that calibrates on tokens generated by the BF16 model in agent sessions. On Qwen3.8-27B, NariQ at 3.0 and 2.5 bits per weight stays level with BF16 on two long-horizon agentic benchmarks, DeepSWE and Terminal Bench, as well as on AIME and RealWorldQA. Our checkpoint at 2.0 bits per weight trails this performance. To our knowledge, these are the first quantizations at or below 3 bits per weight shown to match their full-precision version on long-horizon agentic tasks. We release the files as Qwen3.8-27B-NariQ-GGUF (https://huggingface.co/nari-labs/Qwen3.8-27B-NariQ-GGUF), with inference available through our llama.cpp fork at https://github.com/nari-labs/nari-llama.cpp.

## Channel-wise Vector Quantization: Native 1D Visual Tokenization with Learnable Ordering

DevFeed: [Channel-wise Vector Quantization: Native 1D Visual Tokenization with Learnable Ordering](<https://devfeed.tech/articles/channel-wise-vector-quantization-native-1d-visual-tokenization-with-learnable-ordering-145338.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.26089>)

Author: Wei Song, Tianhang Wang, Yitong Chen, Tong Zhang, Zuxuan Wu, Min Li, Jiaqi Wang, Kaicheng Yu

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [ai-image-generator](<https://devfeed.tech/topics/ai-image-generator.md>), [Image](<https://devfeed.tech/topics/image.md>)

Tags: [autoregressive](<https://devfeed.tech/tags/autoregressive.md>), [channel](<https://devfeed.tech/tags/channel.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [generation](<https://devfeed.tech/tags/generation.md>), [image](<https://devfeed.tech/tags/image.md>), [image-generation](<https://devfeed.tech/tags/image-generation.md>), [ordering](<https://devfeed.tech/tags/ordering.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [scalability](<https://devfeed.tech/tags/scalability.md>), [sequence](<https://devfeed.tech/tags/sequence.md>), [tokenization](<https://devfeed.tech/tags/tokenization.md>)

### AI overview

Channel-wise Vector Quantization (CVQ) tokenizes images by quantizing feature-map channels instead of patch vectors, forming a 1D sequence with learnable coarse-to-fine ordering. Its Channel-wise Autoregressive (CAR) model predicts channels sequentially, first establishing global structure and then adding fine detail. The abstract reports CVQ maintains over 95% codebook utilization up to 2^16 entries, with reconstruction gains over conventional VQ growing as codebook size increases. CAR reports a DPG score of 86.7 and GenEval score of 0.79 for text-to-image generation.

### Source excerpt

arXiv:2605.26089v3 Announce Type: replace-cross Abstract: We present Channel-wise Vector Quantization (CVQ), a novel image tokenization paradigm that replaces patch-wise tokens with channel-wise tokens. Unlike conventional vector quantization, which assigns a discrete token to each patch feature vector, CVQ quantizes each channel of the feature map. This represents an image as a 1D channel sequence with learnable coarse-to-fine ordering, rather than a 2D patch grid. Based on CVQ, we introduce a new visual autoregressive framework with "next-channel prediction". Instead of rendering images patch by patch in raster order, our Channel-wise Autoregressive (CAR) model predicts image channels sequentially, producing progressively enriched visual details. Specifically, it first sketches global structure and then refines fine-grained attributes, akin to a human artist's workflow. Empirically, we show that: (1) CVQ exhibits strong codebook scalability, maintaining 95%+ utilization with up to $2^{16}$ entries while delivering growing reconstruction gains over VQ as the codebook scales up; and (2) CAR attains a DPG score of 86.7 and a GenEval score of 0.79, demonstrating strong effectiveness for text-to-image generation.

## STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization

DevFeed: [STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization](<https://devfeed.tech/articles/stepquant-when-and-where-errors-matter-in-delta-rule-recurrent-state-quantization-144947.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.38169>)

Author: Bingchen Yao, Haobo Xu, Haokun Lin, Yichen Wu, Ziyu Guo, Renrui Zhang, Zhichao Lu, Zhenan Sun, Ying Wei

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [Post-training optimization](<https://devfeed.tech/topics/post-training-optimization.md>), [Error Propagation](<https://devfeed.tech/topics/error-propagation.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [bottleneck](<https://devfeed.tech/tags/bottleneck.md>), [budget](<https://devfeed.tech/tags/budget.md>), [compression](<https://devfeed.tech/tags/compression.md>), [concurrent](<https://devfeed.tech/tags/concurrent.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [delta](<https://devfeed.tech/tags/delta.md>), [dimensions](<https://devfeed.tech/tags/dimensions.md>), [distributions](<https://devfeed.tech/tags/distributions.md>), [error](<https://devfeed.tech/tags/error.md>), [errors](<https://devfeed.tech/tags/errors.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [fixed](<https://devfeed.tech/tags/fixed.md>), [framework](<https://devfeed.tech/tags/framework.md>), [generation](<https://devfeed.tech/tags/generation.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [recurrent](<https://devfeed.tech/tags/recurrent.md>), [state](<https://devfeed.tech/tags/state.md>)

### AI overview

STEPQuant addresses accuracy loss when low-precision quantization errors propagate through Delta-rule recurrent state updates. It accounts for error magnitude and memory lifetime, and jointly fits key-row and value-column scales using state distributions and key-row effects on output error. On Qwen3.8-27B and Kimi-Linear-48B-A3B-Instruct across short- and long-generation benchmarks, it reportedly stays close to FP32-state accuracy at a nominal 6-bit budget and outperforms uniform INT8 in its 4-bit configuration. With optimized GPU kernels in SGLang, its 6-bit version achieves over 5x recurrent-state compression and reduces total serving memory by up to 68.7%.

### Source excerpt

arXiv:2609.38169v2 Announce Type: replace-cross Abstract: Linear attention replaces growing KV caches with fixed-size recurrent states, yet these persistent states can become a substantial memory bottleneck under concurrent serving. Directly quantizing recurrent states to low precision often leads to severe accuracy degradation, as quantization errors propagate through successive state updates. We discover that the impact of these errors depends on two complementary dimensions: temporally, errors in long-lived memory can persist across many decoding steps; spatially, errors in different key rows affect model outputs differently, while state magnitudes vary substantially along both rows and columns. Motivated by these observations, we propose STEPQuant, a spatial-temporal post-training quantization framework for Delta-rule recurrent states. STEPQuant allocates precision according to error magnitude and memory lifetime, and jointly fits key-row and value-column scales based on state distributions and key-row impact on output error. Experiments on Qwen3.8-27B and Kimi-Linear-48B-A3B-Instruct across both long- and short-generation benchmarks show that STEPQuant closely matches FP32-state accuracy under a nominal 6-bit budget and outperforms uniform INT8 in its 4-bit configuration. Integrated into SGLang with optimized GPU kernels, 6-bit STEPQuant achieves over 5x recurrent-state compression and reduces total serving memory by up to 68.7%. Our code is available at https://github.com/Dreamer-Toby/STEPQuant.

## Towards Autoregressive Neural Network Parameter Generation

DevFeed: [Towards Autoregressive Neural Network Parameter Generation](<https://devfeed.tech/articles/towards-autoregressive-neural-network-parameter-generation-145068.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25538>)

Author: Bedionita Soro, Bruno Andreis, Song Chong, Sung Ju Hwang

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [VAE](<https://devfeed.tech/topics/vae.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [autoregressive](<https://devfeed.tech/tags/autoregressive.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [generation](<https://devfeed.tech/tags/generation.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [model](<https://devfeed.tech/tags/model.md>), [modeling](<https://devfeed.tech/tags/modeling.md>), [neural-network](<https://devfeed.tech/tags/neural-network.md>), [parameter](<https://devfeed.tech/tags/parameter.md>), [performance](<https://devfeed.tech/tags/performance.md>)

### AI overview

Instruction-Guided Parameter Generation (IGPG) generates neural network parameters by first quantizing pretrained weights with a VQ-VAE, then training a prefix-conditioned autoregressive model on the resulting tokens. At inference, the VQ-VAE decoder maps generated tokens back into parameter space. The abstract reports competitive in-distribution generation and useful initializations for held-out datasets and architectures; after adaptation, these improve downstream performance over random initialization. The work evaluates in-distribution generation, cross-dataset transfer, and cross-architecture settings, but the supplied text gives no quantitative results or details about the evaluation.

### Source excerpt

arXiv:2611.25538v1 Announce Type: new Abstract: Large Language Models (LLMs) have recently established a new paradigm in machine learning, demonstrating exceptional zero-shot generalization and complex reasoning capabilities. Concurrently, neural network parameter generation has emerged as a promising avenue for rapid task adaptation and model synthesis. Existing weight generation techniques, however, struggle to jointly learn the distribution of diverse models, to encode heterogeneous architectures within a single generator, and to generalize out of distribution. We therefore ask whether pretrained parameters can be encoded into a discrete domain so that the zero-shot generalization LLMs exhibit in text generation can be transferred to weight generation. Following this view, we propose an LLM-inspired approach, Instruction-Guided Parameter Generation (IGPG), that formulates network parameter generation as an autoregressive modeling problem. We first quantize pretrained weights with a VQ-VAE, then train a prefix-conditioned autoregressive generator over the resulting tokens. At inference, the generated weight tokens are mapped back to parameter space by the VQ-VAE decoder. We evaluate the framework across in-distribution generation, cross-dataset transfer, and cross-architecture settings. The proposed method achieves competitive performance in in-distribution generation and provides useful parameter initializations for held-out datasets and architectural configurations, improving downstream performance over random initialization after adaptation. These results suggest that discrete autoregressive modeling provides a promising framework for learning and transferring structure across collections of pretrained neural network parameters.

## Efficient Latent Reasoning for Vision-Language Model via Dynamics-Aware Quantization and Distillation

DevFeed: [Efficient Latent Reasoning for Vision-Language Model via Dynamics-Aware Quantization and Distillation](<https://devfeed.tech/articles/efficient-latent-reasoning-for-vision-language-model-via-dynamics-aware-quantization-and-distillation-145567.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25670>)

Author: Yanjing Li, Sheng Xu, Baochang Zhang, Xianbin Cao, Yawei Li

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [vlm](<https://devfeed.tech/topics/vlm.md>), [Chain-of-thought](<https://devfeed.tech/topics/chain-of-thought.md>), [Post-training optimization](<https://devfeed.tech/topics/post-training-optimization.md>)

Tags: [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [distillation](<https://devfeed.tech/tags/distillation.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [vision](<https://devfeed.tech/tags/vision.md>), [vision-language-model](<https://devfeed.tech/tags/vision-language-model.md>)

### AI overview

QL-VLM aims to make latent chain-of-thought reasoning in vision-language models cheaper without losing predictive accuracy under 4-bit quantization. Its Dynamics-Guided Quantization uses nonuniform codebooks to preserve rotary-key structure for information retrieval; Dynamics-Preserving Distillation aligns continuous thoughts, latent-query readouts, and predictions with a full-precision teacher to support later low-precision reasoning. On ScienceQA-IMG with Qwen2-VL-2B-Instruct, the abstract reports 90.43% accuracy, a 40.33% improvement over baseline, 16.64x fewer bit-operations, and 6.40x lower measured latency than explicit CoT.

### Source excerpt

arXiv:2611.25670v1 Announce Type: new Abstract: Efficient chain-of-thought (CoT) reasoning in vision-language models (VLMs) requires reducing both intermediate reasoning steps and their execution cost. Latent CoT methods compress textual rationales into continuous thoughts, yet their full-precision (FP) models retain substantial storage and computation costs. Lowering numerical precision could reduce these remaining costs, yet direct quantization severely degrades reasoning accuracy, particularly with 4-bit weights and activations. Operator and perturbation analyses reveal key sensitivity and direction-dependent predictive effects of continuous-thought errors, motivating preservation of information retrieval and subsequent reasoning beyond reconstruction accuracy. We therefore introduce QL-VLM, a dynamics-aware Quantization framework for efficient Latent reasoning in Vision-Language Models (VLMs). First, Dynamics-Guided Quantization (DGQ) uses nonuniform codebooks that preserve rotary-key structure, mitigating quantization-induced distortion in vision-language information retrieval. Second, Dynamics-Preserving Distillation (DPD) aligns continuous thoughts, latent-query readouts, and predictions with a full-precision teacher, maintaining predictive consistency through subsequent low-precision reasoning. On ScienceQA-IMG with Qwen2-VL-2B-Instruct, 4-bit QL-VLM achieves 90.43% accuracy and improves baseline by 40.33%, with 16.64x fewer bit-operations and 6.40x lower measured latency than explicit CoT.

## Predicting Lost Updates Is Not Enough to Choose a Rounding Policy

DevFeed: [Predicting Lost Updates Is Not Enough to Choose a Rounding Policy](<https://devfeed.tech/articles/predicting-lost-updates-is-not-enough-to-choose-a-rounding-policy-143289.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2607.09800>)

Author: Zekai Shang

Published: 2026-11-19T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>)

Tags: [32-bit](<https://devfeed.tech/tags/32-bit.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [policy](<https://devfeed.tech/tags/policy.md>), [precision](<https://devfeed.tech/tags/precision.md>), [training](<https://devfeed.tech/tags/training.md>), [validation](<https://devfeed.tech/tags/validation.md>)

### AI overview

In Pythia-410M continuations from a common checkpoint, a higher-precision pilot accurately forecast how often matrix weights stayed unchanged during the first 64 updates, but that prediction did not help select a rounding policy. Across three schedules with optimizer state retained or reset, nearest rounding produced lower validation loss at 256 updates than stochastic rounding or keeping 32-bit weights; the same ordering held at 512 updates when state was retained. These findings are limited to the runs described in the abstract.

### Source excerpt

arXiv:2607.09800v5 Announce Type: replace Abstract: In low-precision training, a weight can round back to its previous value after a small update. We test whether predicting how often this happens helps choose a rounding rule. In Pythia-410M continuations from a common checkpoint, a higher-precision pilot accurately predicted the fraction of matrix weights left unchanged at each of the first 64 updates. Across three schedules with optimizer state retained or reset, nearest rounding gave lower validation loss at 256 updates than stochastic rounding or retaining 32-bit weights. The same ordering held at 512 updates with retained state. Accurately forecasting the unchanged fraction did not identify a benefit from switching policies in these runs.

## Same Model, Different People: Numerical Execution Contracts and the Reproducibility of Fairness Audits

DevFeed: [Same Model, Different People: Numerical Execution Contracts and the Reproducibility of Fairness Audits](<https://devfeed.tech/articles/same-model-different-people-numerical-execution-contracts-and-the-reproducibility-of-fairness-audits-143426.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23657>)

Author: Erzheng Zhang, Agnes Chu

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [reproducibility](<https://devfeed.tech/topics/reproducibility.md>), [ML model versioning](<https://devfeed.tech/topics/ml-model-versioning.md>)

Tags: [audits](<https://devfeed.tech/tags/audits.md>), [contracts](<https://devfeed.tech/tags/contracts.md>), [cs-cy](<https://devfeed.tech/tags/cs-cy.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [execution](<https://devfeed.tech/tags/execution.md>), [fairness](<https://devfeed.tech/tags/fairness.md>), [model](<https://devfeed.tech/tags/model.md>), [reproducibility](<https://devfeed.tech/tags/reproducibility.md>)

### AI overview

Fairness audit results can depend on the numerical execution configuration, even when the model, records, threshold and audit code are fixed. With byte-identical weights, changing arithmetic altered some near-threshold decisions, but no prespecified unweighted verdict; reduced-precision deployment configurations changed more decisions and some diagnostic verdicts, but no four-fifths verdict. In one example, vLLM FP8 serving of a 24B model selected 54% of people versus 45% for strict FP32 and passed an employment-status audit by sex that FP32 failed. At FP32's selection count, FP8 failed demographic parity, and the equalized-odds change had no paired interval excluding zero; the result does not establish that FP8 is fairer. Every changed pass/fail verdict had a sampling interval containing its criterion. The proposed stability protocol declares execution contracts, groups, weights and tests before scoring, then reports verdict variation separately from sampling uncertainty.

### Source excerpt

arXiv:2611.23657v1 Announce Type: cross Abstract: Fairness audits treat a trained model as a fixed decision function, but the arithmetic that evaluates it is part of that function. This study holds the model, records, threshold and audit code fixed and varies the numerical execution contract in two tiers. Tier 1 changes only how products and sums are computed, with the serialized weights byte-identical, and is the causal test of arithmetic. Tier 2 runs reduced-precision deployment configurations (BF16 compute, FP8 and 8-bit weights), which can also round weights and change engines and kernels, so its effects belong to the configuration. It audits three tabular model families and two open-weight language models (4B and 24B parameters) on four public tasks, including hospital readmission, on CPUs and GPUs from NVIDIA, AMD and Apple. Tier 1 changed decisions near the threshold, and which ones depended on the GPU and its software stack, but no verdict under the prespecified, unweighted tests. Tier 2 configurations changed more decisions and some diagnostic verdicts, but no four-fifths verdict. Serving the 24B model with vLLM FP8 selected 54% of people rather than 45% and passed an employment-status audit by sex that its strict FP32 reference failed: its demographic-parity and equalized-odds gaps fell from 0.0567 and 0.0515 to 0.0417 and 0.0394, against a diagnostic tolerance of 0.05. At the reference's selection count FP8 failed demographic parity, and no reading gave its equalized-odds change a paired interval excluding zero. The declared audit depends on the configuration; FP8 is not shown to be fairer. At the frozen thresholds, every pass-fail verdict a configuration changed had a sampling interval containing its criterion. It proposes an audit stability protocol in which the auditor declares contracts, groups, weights and tests before scoring and records each verdict's range over them apart from its sampling uncertainty.

## Perplexity Ranks Activation Quantisers but Understates What They Remove

DevFeed: [Perplexity Ranks Activation Quantisers but Understates What They Remove](<https://devfeed.tech/articles/perplexity-ranks-activation-quantisers-but-understates-what-they-remove-143312.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.23125>)

Author: Anish Sathyanarayanan, Min Xu

Published: 2026-11-19T05:00:00Z

Content type: opinion

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>)

Tags: [activation](<https://devfeed.tech/tags/activation.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [error](<https://devfeed.tech/tags/error.md>), [factor](<https://devfeed.tech/tags/factor.md>), [full](<https://devfeed.tech/tags/full.md>), [in-between](<https://devfeed.tech/tags/in-between.md>), [outlier](<https://devfeed.tech/tags/outlier.md>), [perplexity](<https://devfeed.tech/tags/perplexity.md>), [qa](<https://devfeed.tech/tags/qa.md>), [qwen3](<https://devfeed.tech/tags/qwen3.md>), [reporting](<https://devfeed.tech/tags/reporting.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [rotation](<https://devfeed.tech/tags/rotation.md>)

### AI overview

Across four language models, perplexity usually ranks four-bit activation quantisers in the same order as closed-book QA loss (33 of 34 setting pairs), but understates the damage: a 6-16% perplexity increase accompanies a 10-23% accuracy drop, with rare facts most affected. This alignment holds when quantisation error behaves like random noise; without rotation, rounding small values to zero removes signal and harms in-context retrieval more. Input crest factor helps distinguish these cases. Blockwise rotation can leave outliers when model width lacks a ready-made Hadamard matrix; the authors report this for 10 of 42 popular open models. They say full-width Hadamard rotations are available for those widths at about the same cost, and on Qwen3-4B reduced closed-book TriviaQA loss from 0.112 to 0.060 across three seeds. They recommend reporting a knowledge-heavy task alongside perplexity and using full-width rotation when blocks leave outlier-heavy inputs.

### Source excerpt

arXiv:2609.23125v2 Announce Type: replace Abstract: Four-bit activation quantisation of language models is usually tuned and compared by perplexity. We ask what that misses, and find three things. First, perplexity gets the order of quantisers right but the size of their damage wrong. Across four models it ranks 33 of 34 pairs of settings correctly by closed-book QA loss, yet a 6-16% rise in perplexity comes with a 10-23% drop in QA accuracy, and the lost facts are mostly rare ones. Second, the ranking only holds while the quantisation error acts like random noise. Without rotation, rounding small values to zero deletes part of the signal, and this hurts in-context retrieval far more than noise of the same size would. The crest factor of the quantiser's input (its peak-to-average ratio) helps tell the two cases apart. Third, rotating in blocks, the usual workaround when a layer's width has no ready-made Hadamard matrix, can land a model in between and leave outliers behind; 10 of 42 popular open models have such a width. A full-width Hadamard exists for every such width we found, costs about the same, and on Qwen3-4B halves the closed-book TriviaQA loss (0.112 to 0.060, over three rotation seeds). We recommend reporting a knowledge-heavy task alongside perplexity, and rotating at full width wherever blocks leave an input outlier-heavy.

## Zonotopic Analysis of Quantization Error in Neural Networks

DevFeed: [Zonotopic Analysis of Quantization Error in Neural Networks](<https://devfeed.tech/articles/zonotopic-analysis-of-quantization-error-in-neural-networks-143543.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24648>)

Author: Robert Debbas, \\'Eric Goubault, Sylvie Putot

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [analysis](<https://devfeed.tech/tags/analysis.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [convex-hull](<https://devfeed.tech/tags/convex-hull.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-lo](<https://devfeed.tech/tags/cs-lo.md>), [dra](<https://devfeed.tech/tags/dra.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [error](<https://devfeed.tech/tags/error.md>), [faster](<https://devfeed.tech/tags/faster.md>), [framework](<https://devfeed.tech/tags/framework.md>), [improvements](<https://devfeed.tech/tags/improvements.md>), [linear](<https://devfeed.tech/tags/linear.md>), [network](<https://devfeed.tech/tags/network.md>), [neural-networks](<https://devfeed.tech/tags/neural-networks.md>), [precision](<https://devfeed.tech/tags/precision.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [relu](<https://devfeed.tech/tags/relu.md>), [runtimes](<https://devfeed.tech/tags/runtimes.md>), [transformer](<https://devfeed.tech/tags/transformer.md>), [verification](<https://devfeed.tech/tags/verification.md>)

### AI overview

The paper presents a zonotope-based method for soundly bounding quantization error in ReLU feed-forward networks, with a unified transformer for rounding and clamping. Its constrained zonotope version adds linear constraints and, on ACAS Xu at perturbation radii r=3 and r=6, reduced average error-bound widths by about 70-85% and runtimes by 65-80% versus QEBVerif's symbolic DRA propagation. The gains varied by bitwidth; symbolic DRA produced tighter bounds at larger radii. Plain zonotopes gave competitive sound bounds nearly 300 times faster on average.

### Source excerpt

arXiv:2611.24648v1 Announce Type: new Abstract: Quantized neural networks achieve computational efficiency by reducing precision, but bounding their quantization error remains a challenge due to discrete rounding and clamping operations. We present a zonotope-based abstract interpretation framework for quantized ReLU feed-forward network verification, introducing a unified round-clamp abstract transformer that jointly handles both operations. We extend this framework to constrained zonotopes, which add linear constraints to tighten bounds, and derive sound quantization error representations. We establish soundness and affine-strip optimality of the round-clamp abstract transformer, as well as convex-hull optimality of its complete constrained formulation. On the ACAS Xu benchmarks, our constrained zonotope method reduces average error-bound widths by approximately 70-85% and runtimes by 65-80% compared to QEBVerif's symbolic DRA propagation at the two smallest tested perturbation radii (r=3 and r=6). These improvements are not uniform across bitwidths, and symbolic DRA achieves tighter bounds at larger radii. Our plain zonotope method delivers competitive sound bounds nearly 300 times faster on average.

## ICICLE: In-Context Inference for Curve-Learning Extrapolation

DevFeed: [ICICLE: In-Context Inference for Curve-Learning Extrapolation](<https://devfeed.tech/articles/icicle-in-context-inference-for-curve-learning-extrapolation-143527.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24469>)

Author: Laurits Fredsgaard, Michael Riis Andersen, Mikkel N. Schmidt

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [onnx](<https://devfeed.tech/topics/onnx.md>)

Tags: [bayesian](<https://devfeed.tech/tags/bayesian.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [budget](<https://devfeed.tech/tags/budget.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [encoder](<https://devfeed.tech/tags/encoder.md>), [extrapolation](<https://devfeed.tech/tags/extrapolation.md>), [freeze](<https://devfeed.tech/tags/freeze.md>), [inference](<https://devfeed.tech/tags/inference.md>), [jax](<https://devfeed.tech/tags/jax.md>), [learning](<https://devfeed.tech/tags/learning.md>), [meta](<https://devfeed.tech/tags/meta.md>), [model](<https://devfeed.tech/tags/model.md>), [noise](<https://devfeed.tech/tags/noise.md>), [number](<https://devfeed.tech/tags/number.md>), [onnx](<https://devfeed.tech/tags/onnx.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [pytorch](<https://devfeed.tech/tags/pytorch.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [ray](<https://devfeed.tech/tags/ray.md>), [release](<https://devfeed.tech/tags/release.md>)

### AI overview

ICICLE is an in-context surrogate for freeze-thaw Bayesian optimization that accepts a variable number of hyperparameters through a shared set encoder. Trained on synthetic tasks covering typed hyperparameters, heavy-tailed noise, quantization, and divergence, it uses 1.28 million parameters (8.7% of FT-PFN's count) and produces similar mean-regret trajectories to FT-PFN on LCBench, PD1, and TaskSet. On Meta-Album's 86 tasks with 33 hyperparameters, it achieves lower mean normalized regret than Quick-Tune overall and on extended, with no clear difference on mini or micro. ICICLE was not pre-trained or meta-trained on Meta-Album tasks, while Quick-Tune was meta-trained on related tasks. The release samples synthetic tasks on the fly and includes a Ray Tune freeze-thaw scheduler and an ONNX inference model that does not require JAX or PyTorch.

### Source excerpt

arXiv:2611.24469v1 Announce Type: new Abstract: Freeze-thaw Bayesian optimization allocates training budget by predicting how partially observed learning curves will continue. LC-PFN first applied prior-data fitted networks (PFNs) to learning-curve extrapolation, and FT-PFN extended this approach to configuration-aware freeze-thaw optimization with a ten-slot hyperparameter encoder. We introduce In-Context Inference for Curve-Learning Extrapolation (ICICLE), an in-context surrogate that accepts a variable number of hyperparameters through a shared set encoder and is trained on a synthetic prior covering typed hyperparameters, heavy-tailed noise, quantization, and divergence. ICICLE has 1.28 million parameters (8.7% of FT-PFN's count) and produces similar mean-regret trajectories to FT-PFN on LCBench, PD1, and TaskSet. On the 86-task Meta-Album benchmark with 33 hyperparameters, ICICLE reaches lower mean normalized regret than Quick-Tune overall and on extended, with no clear difference on mini or micro. It does so without pre-training or meta-training on Meta-Album tasks, whereas Quick-Tune is meta-trained on related Meta-Album tasks. Our release samples synthetic training tasks on the fly and provides a freeze-thaw scheduler for Ray Tune and an ONNX model for inference without JAX or PyTorch.

## CoreQ: Mismatch-Corrected Successive Rounding for LLM Quantization

DevFeed: [CoreQ: Mismatch-Corrected Successive Rounding for LLM Quantization](<https://devfeed.tech/articles/coreq-mismatch-corrected-successive-rounding-for-llm-quantization-142504.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2602.05902>)

Author: Seohyeon Cha, Huancheng Chen, Dongjun Kim, Haoran Zhang, Kevin Chan, Gustavo de Veciana, Haris Vikalo

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [Post-training optimization](<https://devfeed.tech/topics/post-training-optimization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [beam](<https://devfeed.tech/tags/beam.md>), [beam-search](<https://devfeed.tech/tags/beam-search.md>), [bounded](<https://devfeed.tech/tags/bounded.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [framework](<https://devfeed.tech/tags/framework.md>), [github](<https://devfeed.tech/tags/github.md>), [greedy](<https://devfeed.tech/tags/greedy.md>), [layer](<https://devfeed.tech/tags/layer.md>), [layers](<https://devfeed.tech/tags/layers.md>), [learning](<https://devfeed.tech/tags/learning.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-quantization](<https://devfeed.tech/tags/llm-quantization.md>), [overfitting](<https://devfeed.tech/tags/overfitting.md>), [performance](<https://devfeed.tech/tags/performance.md>), [perplexity](<https://devfeed.tech/tags/perplexity.md>), [post-training](<https://devfeed.tech/tags/post-training.md>), [precision](<https://devfeed.tech/tags/precision.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [search](<https://devfeed.tech/tags/search.md>), [target](<https://devfeed.tech/tags/target.md>)

### AI overview

CoreQ addresses activation mismatch in sequential post-training quantization, where earlier layers' quantization errors change the inputs to later layers. It uses a geometric decomposition to derive a closed-form, layer-adaptive coefficient that limits how much empirical mismatch shifts each calibration target, aiming to reduce overfitting on small calibration sets without coefficient tuning. It then applies greedy successive rounding to a triangular least-squares objective. K-CoreQ adds bounded beam search, trading modest extra compute for reported performance gains. Across multiple LLM families, scales, bit-widths and quantization settings, the abstract reports lower perplexity and higher downstream accuracy than strong PTQ baselines.

### Source excerpt

arXiv:2602.05902v3 Announce Type: replace-cross Abstract: Post-training quantization (PTQ) enables efficient deployment of large language models by mapping pretrained weights to low-bit formats without retraining, typically using a small calibration set to minimize a layer-wise calibration objective. However, this sequential procedure induces a mismatch: errors from earlier quantized layers alter the inputs received by later layers, causing the activations to deviate from those of the full-precision model. Recent approaches introduce mismatch-aware calibration objectives to compensate for this effect, but leave open how much of the observed mismatch should shift each layer's calibration target. We show that fully applying the empirical mismatch correction can overfit finite calibration data, while using a fixed correction coefficient ignores variation in mismatch estimates across layers. To address these limitations, we propose CoreQ, a learning-free PTQ framework that uses a geometric decomposition of the mismatch to compute a closed-form, layer-adaptive correction coefficient. This coefficient controls how much of the observed mismatch enters each layer's calibration target, reducing overfitting to limited calibration data without coefficient tuning. CoreQ then applies efficient greedy successive rounding to the resulting triangular least-squares objective. Its bounded beam-search extension, K-CoreQ, trades modest additional compute for improved performance. Across multiple LLM families, scales, bit-widths, and quantization settings, CoreQ and K-CoreQ achieve lower perplexity and higher downstream accuracy than strong PTQ baselines. Code is available at GitHub.

## SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference

DevFeed: [SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference](<https://devfeed.tech/articles/sharq-bridging-activation-sparsity-and-fp4-quantization-for-llm-inference-142535.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2606.26587>)

Author: Haoqian Meng, Yilun Luo, Yafei Zhao, Wenyuan Liu, Huaqing Zheng, Xindian Ma, Peng Zhang

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>)

Tags: [accelerators](<https://devfeed.tech/tags/accelerators.md>), [activation](<https://devfeed.tech/tags/activation.md>), [adaptive](<https://devfeed.tech/tags/adaptive.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [block](<https://devfeed.tech/tags/block.md>), [bridging](<https://devfeed.tech/tags/bridging.md>), [code](<https://devfeed.tech/tags/code.md>), [compression](<https://devfeed.tech/tags/compression.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [error](<https://devfeed.tech/tags/error.md>), [floating-point](<https://devfeed.tech/tags/floating-point.md>), [formats](<https://devfeed.tech/tags/formats.md>), [fp16](<https://devfeed.tech/tags/fp16.md>), [fp4](<https://devfeed.tech/tags/fp4.md>), [fp8](<https://devfeed.tech/tags/fp8.md>), [inference](<https://devfeed.tech/tags/inference.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [sparsity](<https://devfeed.tech/tags/sparsity.md>)

### AI overview

SharQ combines activation sparsity with FP4 quantization using an online sparse-dense decomposition. It selects an input-adaptive N:M mask for an outlier-heavy sparse backbone, quantizes that backbone to FP4, then builds a dense residual against the quantized backbone. Sparse and dense FP4 GEMMs process the two paths, with the dense path compensating for mask loss and sparse-path quantization error. The paths share one FP4 weight payload with separate scale views; a fused preparation kernel combines mask generation, residual construction, and layer normalization. The method requires no calibration, retraining, or model-specific tuning. Across four listed language and vision-language models, it reportedly recovers 43-63% of the NVFP4-to-FP16 accuracy gap and generalizes across NVFP4, HiF4, and MXFP4. On an RTX 5090, reported gains are 2.2-2.4x lower latency than FP16 and 1.2-1.4x higher throughput than FP8 for language serving; with SageAttention, it reaches up to 1.58x speedup on Wan2.2 video generation.

### Source excerpt

arXiv:2606.26587v2 Announce Type: replace-cross Abstract: Low-bit floating-point formats and semi-structured sparsity are increasingly supported by modern accelerators, yet combining them for LLM activation compression remains challenging: activations contain input-dependent outliers that dominate block scales in FP4 quantization, and directly applying N:M sparsity masks discards moderate values, coupling sparsification loss with quantization error. We introduce SharQ, a training-free inference method that bridges activation sparsity and FP4 quantization through an online sparse--dense decomposition. For each activation tensor, SharQ generates an input-adaptive N:M mask to extract an outlier-dominated sparse backbone, quantizes it to FP4, and defines a dense residual relative to the quantized sparse backbone rather than the unquantized sparse values. A sparse FP4 GEMM processes the backbone while a dense FP4 GEMM compensates for both mask-induced activation loss and sparse-path quantization error. The two paths share a single FP4 weight payload with path-specific scale views, and a fused preparation kernel absorbs mask generation, residual construction, and layer normalization into one operator. SharQ requires no calibration data, retraining, or model-specific tuning. Evaluated on Llama-3.1-8B, Qwen2.5-7B, Qwen3-30B-A3B, and Qwen3-VL-8B, SharQ recovers 43--63% of the NVFP4-to-FP16 accuracy gap across language and vision-language tasks, and generalizes across NVFP4, HiF4, and MXFP4 formats. On an RTX 5090, SharQ delivers 2.2--2.4$\times$ latency reduction over FP16 and 1.2--1.4$\times$ throughput improvement over FP8 in language model serving, and up to 1.58$\times$ speedup on Wan2.2-T2V-A14B video generation when combined with SageAttention. Our code is available at https://github.com/actypedef/SharQ.

## EEG2MCU: Sensing-to-Execution Co-Design for Wearable EEG Inference

DevFeed: [EEG2MCU: Sensing-to-Execution Co-Design for Wearable EEG Inference](<https://devfeed.tech/articles/eeg2mcu-sensing-to-execution-co-design-for-wearable-eeg-inference-143417.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23619>)

Author: Shoma Yukawa, Meghna Roychowdhury, Arnab Sanyal, Chengwei Zhou, Shreyas Sen, Gourav Datta

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [Local AI](<https://devfeed.tech/topics/local-ai.md>)

Tags: [co-design](<https://devfeed.tech/tags/co-design.md>), [cortex-m7](<https://devfeed.tech/tags/cortex-m7.md>), [cs-ar](<https://devfeed.tech/tags/cs-ar.md>), [cs-hc](<https://devfeed.tech/tags/cs-hc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data-movement](<https://devfeed.tech/tags/data-movement.md>), [eeg](<https://devfeed.tech/tags/eeg.md>), [eess-sp](<https://devfeed.tech/tags/eess-sp.md>), [embedded](<https://devfeed.tech/tags/embedded.md>), [energy](<https://devfeed.tech/tags/energy.md>), [flash](<https://devfeed.tech/tags/flash.md>), [halo](<https://devfeed.tech/tags/halo.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [hardware-design](<https://devfeed.tech/tags/hardware-design.md>), [inference](<https://devfeed.tech/tags/inference.md>), [int8](<https://devfeed.tech/tags/int8.md>), [latency](<https://devfeed.tech/tags/latency.md>), [materialization](<https://devfeed.tech/tags/materialization.md>), [mcu](<https://devfeed.tech/tags/mcu.md>), [memory](<https://devfeed.tech/tags/memory.md>), [reduce](<https://devfeed.tech/tags/reduce.md>), [reduction](<https://devfeed.tech/tags/reduction.md>), [sensing](<https://devfeed.tech/tags/sensing.md>), [wearable](<https://devfeed.tech/tags/wearable.md>)

### AI overview

EEG2MCU reduces wearable EEG inference costs on a Cortex-M7 by co-designing electrode selection and execution. It uses task-aware montage selection and generates only the INT8 input regions needed for each first-layer tile, with halo reuse and access-aware placement across Flash, SRAM, and tightly coupled memory to limit redundant generation and data movement. Compared with the baseline, it reports 11.38x lower peak SRAM (including a 1.71x reduction from selecting 19 rather than 62 electrodes), 2.57x lower latency, and 2.48x lower energy, while retaining 94-96% of full-montage accuracy.

### Source excerpt

arXiv:2611.23619v1 Announce Type: new Abstract: Wearable EEG inference is constrained not only by model size, but also by the channel-dependent sensing front end that must process high-dimensional inputs before compact downstream computation. We present EEG2MCU, a sensing-to-execution co-design framework that treats electrode count as a hardware design variable through task-aware montage selection. EEG2MCU retains a compact time-frequency representation and generates only the INT8 regions required by each first-layer tile, avoiding full input materialization. Halo reuse and access-aware placement across Flash, SRAM, and tightly coupled memory further reduce redundant generation and data movement. On a Cortex-M7 MCU, EEG2MCU reduces peak SRAM by 11.38x relative to the baseline, including a 1.71x reduction from the 62-to-19 electrode montage, while reducing latency by 2.57x and energy by 2.48x. It retains 94-96% of full-montage accuracy, demonstrating practical high-dimensional EEG inference under tight embedded-resource constraints.

## MoMoVL: Modality-Aware Adaptive Compression for On-Device Sparsely-Activated Vision-Language Models

DevFeed: [MoMoVL: Modality-Aware Adaptive Compression for On-Device Sparsely-Activated Vision-Language Models](<https://devfeed.tech/articles/momovl-modality-aware-adaptive-compression-for-on-device-sparsely-activated-vision-language-models-143435.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23695>)

Author: Dongkeun Lee, Wonjun Lee

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [vlm](<https://devfeed.tech/topics/vlm.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>)

Tags: [activation](<https://devfeed.tech/tags/activation.md>), [adaptive](<https://devfeed.tech/tags/adaptive.md>), [compression](<https://devfeed.tech/tags/compression.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [device](<https://devfeed.tech/tags/device.md>), [edge](<https://devfeed.tech/tags/edge.md>), [edge-devices](<https://devfeed.tech/tags/edge-devices.md>), [inference](<https://devfeed.tech/tags/inference.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [memory](<https://devfeed.tech/tags/memory.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [moe](<https://devfeed.tech/tags/moe.md>), [on-device](<https://devfeed.tech/tags/on-device.md>), [precision](<https://devfeed.tech/tags/precision.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [sparsity](<https://devfeed.tech/tags/sparsity.md>), [tokens](<https://devfeed.tech/tags/tokens.md>), [trade](<https://devfeed.tech/tags/trade.md>), [vision](<https://devfeed.tech/tags/vision.md>), [vision-language-models](<https://devfeed.tech/tags/vision-language-models.md>)

### AI overview

MoMoVL assigns mixed-precision bit-widths in MoE vision-language models using both per-expert activation profiles and each modality's quantization sensitivity. The approach accounts for differences between visual and text tokens, which can affect expert quantization. The abstract reports a better accuracy-compression trade-off than state-of-the-art mixed-precision baselines and up to 15.44% lower memory use than a uniform-precision baseline, with a modest accuracy drop. The abstract does not specify evaluation details or the size of that drop.

### Source excerpt

arXiv:2611.23695v1 Announce Type: cross Abstract: The rise of the agentic web has been accompanied by growing demand for on-device Vision-Language Models (VLMs) that perform diverse cross-modal web tasks with enhanced privacy and reduced latency. Mixture-of-Experts (MoE) models are widely adopted in modern VLMs for compute-efficient scalability, yet their multi-expert architecture poses deployment challenges due to substantial memory demands. Mixed-precision quantization is an effective compression technique that reduces bit-widths while adapting to the heterogeneous hardware budgets of edge devices, particularly well-suited for MoE due to its inherent structural sparsity, which causes imbalanced importance among experts. While prior work has applied quantization for efficient MoE inference at the edge, we point out that in MoE-based VLMs, the modality-specific characteristics of visual and text tokens further impact expert quantization. To address this issue, we present MoMoVL, a modality-aware adaptive compression framework for MoE-based VLMs that brings vision-language inference within reach of on-device resource budgets. Motivated by our empirical findings on modality-induced disparities across experts, MoMoVL jointly exploits the per-expert activation profile and per-modality quantization sensitivity to guide bit-width assignment. Experimental results show that MoMoVL improves the accuracy-compression trade-off relative to state-of-the-art mixed-precision baselines and reduces memory footprint by up to 15.44% with a modest drop in accuracy over its uniform-precision baseline. Our code and data are publicly available at https://github.com/sco-edge/MoMoVL.

## Implementing Binary Transformers with Approximate Memories with Permutation Invariance Enabled Selective (PIES) Protection

DevFeed: [Implementing Binary Transformers with Approximate Memories with Permutation Invariance Enabled Selective (PIES) Protection](<https://devfeed.tech/articles/implementing-binary-transformers-with-approximate-memories-with-permutation-invariance-enabled-selective-pies-protection-142645.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23358>)

Author: Zhen Gao, Xiao Yin, Kexin Liang, Pedro Reviriego, Shanshan Liu, Fabrizio Lombardi

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>), [bert](<https://devfeed.tech/topics/bert.md>), [ecc ram](<https://devfeed.tech/topics/ecc-ram.md>)

Tags: [bert](<https://devfeed.tech/tags/bert.md>), [complexity](<https://devfeed.tech/tags/complexity.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [devices](<https://devfeed.tech/tags/devices.md>), [edge-devices](<https://devfeed.tech/tags/edge-devices.md>), [error](<https://devfeed.tech/tags/error.md>), [error-correction](<https://devfeed.tech/tags/error-correction.md>), [invariance](<https://devfeed.tech/tags/invariance.md>), [memory](<https://devfeed.tech/tags/memory.md>), [model](<https://devfeed.tech/tags/model.md>), [overhead](<https://devfeed.tech/tags/overhead.md>), [performance](<https://devfeed.tech/tags/performance.md>), [permutation](<https://devfeed.tech/tags/permutation.md>), [permutations](<https://devfeed.tech/tags/permutations.md>), [power](<https://devfeed.tech/tags/power.md>), [protection](<https://devfeed.tech/tags/protection.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [recover](<https://devfeed.tech/tags/recover.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [transformers](<https://devfeed.tech/tags/transformers.md>)

### AI overview

The study evaluates how memory bit errors affect binary Transformers, using BERT as a case study. It reports large accuracy losses only at high bit error rates, such as 5% or 10%. The proposed PIES scheme uses Transformers' invariance to row and column permutations to identify critical weights and selectively protect them. Simulations indicate that PIES recovers most performance loss at error rates up to 10%, with substantially lower overhead than traditional error correction schemes. These findings suggest binary Transformers may be feasible with approximate memory and aggressive voltage scaling on low-power edge devices.

### Source excerpt

arXiv:2611.23358v1 Announce Type: cross Abstract: Quantization is an effective technique to reduce memory storage, computation complexity and power for Transformers, and the binarization of the weights is important for meeting these requirements. Approximate memory can further reduce the power for memory, but it also introduces errors. Current research for binary Transformers focuses on task performance, but the model dependable operation against errors on the memory has not been well studied. In this paper, we first evaluate the dependability of binary Transformers with the Bidirectional Encoder Representations from Transformers (BERT) as a case study. A large accuracy loss occurs only at high Bit Error Rates (e.g. BER = 5% or 10%). Then we propose a novel Permutation Invariance Enabled Selective (PIES) scheme to protect the binary weights at very low overhead. This scheme exploits the invariance of Transformers to permutations in the rows and columns to group the critical weights and perform selective protection on those weights. Simulation results show that PIES can recover most of the performance loss caused by bit errors at BERs as high as 10%, at an overhead significantly lower than traditional error correction schemes. These results confirm the feasibility of running binary Transformers using approximate memory with aggressive voltage scaling on resource-limited low power edge devices. The code for PIES is released on github (https://github.com/YangZhutian/binary-transformer-prefix-repair).

## Why Sub-4-Bit Vision CNNs Collapse and Where Mixed-Precision Recovery Stops

DevFeed: [Why Sub-4-Bit Vision CNNs Collapse and Where Mixed-Precision Recovery Stops](<https://devfeed.tech/articles/why-sub-4-bit-vision-cnns-collapse-and-where-mixed-precision-recovery-stops-142655.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23445>)

Author: Michael Schroeder

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>)

Tags: [architectures](<https://devfeed.tech/tags/architectures.md>), [arm](<https://devfeed.tech/tags/arm.md>), [batch](<https://devfeed.tech/tags/batch.md>), [bits](<https://devfeed.tech/tags/bits.md>), [cnns](<https://devfeed.tech/tags/cnns.md>), [compression](<https://devfeed.tech/tags/compression.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [edge](<https://devfeed.tech/tags/edge.md>), [efficientnet](<https://devfeed.tech/tags/efficientnet.md>), [execution](<https://devfeed.tech/tags/execution.md>), [floating-point](<https://devfeed.tech/tags/floating-point.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [int4](<https://devfeed.tech/tags/int4.md>), [int8](<https://devfeed.tech/tags/int8.md>), [layer](<https://devfeed.tech/tags/layer.md>), [less](<https://devfeed.tech/tags/less.md>), [precision](<https://devfeed.tech/tags/precision.md>), [recovery](<https://devfeed.tech/tags/recovery.md>), [vision](<https://devfeed.tech/tags/vision.md>)

### AI overview

The study reports that sub-4-bit quantization causes nonlinear accuracy loss in Vision CNNs, concentrated in late-stage pointwise projections and depthwise layers. On ResNet and EfficientNet backbones evaluated against the full ImageNet-1k validation set, retaining 8-bit precision for under 5% of critical layer activations recovered up to 92% of the accuracy lost. But smaller model storage did not guarantee faster CPU inference: software bit-unpacking slowed execution by 1.48x to 21.5x on ARM and x86, depending on batch size. The results indicate a trade-off between model footprint and throughput; the abstract does not specify where recovery stops.

### Source excerpt

arXiv:2611.23445v1 Announce Type: new Abstract: Deep vision models operating in resource-constrained environments depend heavily on low-bit quantization, yet compressing weights below 4 bits frequently induces catastrophic accuracy collapse. While mixed-precision recovery strategies aim to rescue accuracy by selectively retaining high-precision parameters, the fundamental mechanics of where and why these models collapse remain poorly understood. In this study, we systematically analyze the anatomy of quantization collapse across standard Vision CNNs (including ResNet and EfficientNet backbones) evaluated on the full 50,000-image ImageNet-1k validation set. First, we map the exact progression of degradation, demonstrating that collapse is highly non-linear and concentrated in late-stage pointwise projections and depthwise layers. Second, we introduce a structured, multi-step recovery waterfall that evaluates selective integer precision (INT4/INT8) and selective floating-point scales. We show that keeping less than 5% of critical layer activations in 8-bit precision recovers up to 92% of the accuracy lost during aggressive sub-4-bit compression. Finally, we bridge the hardware reality gap by evaluating deployment performance on standard ARM and x86 CPU architectures. We show that while low-bit quantization reduces memory storage requirements, standard runtime software bit-unpacking introduces a 1.48x to 21.5x execution slowdown depending on batch size, highlighting a critical trade-off between model footprint and actual edge throughput. Our findings provide a practical blueprint for navigating the limits of sub-4-bit deployment without sacrificing hardware performance.

[Next page](<https://devfeed.tech/topics/quantization.md?cursor=WyIyMDI2LTExLTE5VDA1OjAwOjAwKzAwOjAwIiwgIjZhZWE4ZTEyLWRlZGEtNDdiNi1iMjk5LTAwMjEwMDAwODExMiJd>)