# Faster Dynamically Quantized Inference with XNNPack

DevFeed: [Faster Dynamically Quantized Inference with XNNPack](<https://devfeed.tech/articles/faster-dynamically-quantized-inference-with-xnnpack-7410.md>)

Original publisher: [Read original article](<https://blog.tensorflow.org/2024/04/faster-dynamically-quantized-inference-with-xnnpack.html>)

Author: TensorFlow Blog (noreply@blogger.com)

Published: 2024-04-09T16:00:00Z

Content type: article

Language: en

Sources: [The TensorFlow Blog](<https://devfeed.tech/sources/the-tensorflow-blog.md>)

Topics: [Inference Performance](<https://devfeed.tech/topics/inference-performance.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [TensorFlow Lite](<https://devfeed.tech/topics/tensorflow-lite.md>), [cpu](<https://devfeed.tech/topics/cpu.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [model-deployment](<https://devfeed.tech/topics/model-deployment.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [announcement](<https://devfeed.tech/tags/announcement.md>), [article](<https://devfeed.tech/tags/article.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [inference](<https://devfeed.tech/tags/inference.md>), [inference-performance](<https://devfeed.tech/tags/inference-performance.md>), [learn](<https://devfeed.tech/tags/learn.md>), [ml](<https://devfeed.tech/tags/ml.md>), [performance](<https://devfeed.tech/tags/performance.md>), [precision](<https://devfeed.tech/tags/precision.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [range](<https://devfeed.tech/tags/range.md>), [scale](<https://devfeed.tech/tags/scale.md>), [tensorflow](<https://devfeed.tech/tags/tensorflow.md>), [tensorflow-lite](<https://devfeed.tech/tags/tensorflow-lite.md>), [tensors](<https://devfeed.tech/tags/tensors.md>)

## AI overview

This article explains how dynamic range quantization for XNNPack's Fully Connected and Convolution 2D operators improves CPU inference performance in TensorFlow Lite. The approach quantizes weights to 8-bit integers during model conversion and dynamically quantizes activations during inference, delivering most of the performance benefits of full quantization while retaining higher overall accuracy. The article reports a fourfold performance improvement over the single-precision baseline and describes how this can enable AI-powered features on older and lower-tier devices.

## Source excerpt

Posted by Alan Kelly, Software Engineer We are excited to announce that XNNPack's Fully Connected and Convolution 2D operators now support dynamic range quantization. XNNPack is TensorFlow Lite's CPU backend and CPUs deliver the widest reach for ML inference and remain the default target for TensorFlow Lite. Consequently, improving CPU inference performance is a top priority. We quadrupled inference performance in TensorFlow Lite's XNNPack backend compared to the single precision baseline by adding support for dynamic range quantization to the Fully Connected and Convolution operators. This means that more AI powered features may be deployed to older and lower tier devices. Previously, XNNPack offered users the choice between either full integer quantization, where the weights and activations are stored as signed 8-bit integers, or half-precision (fp16) or single-precision (fp32) floating-point inference. In this article we demonstrate the benefits of dynamic range quantization. Dynamic Range Quantization Dynamically quantized models are similar to fully-quantized models in that the weights for the Fully Connected and Convolution operators are quantized to 8-bit integers during model conversion. All other tensors are not quantized, they remain as float32 tensors. During model inference, the floating-point layer activations are converted to 8-bit integers before being passed to the Fully Connected and Convolution operators. The quantization parameters (the zero point and scale) for each row of the activation tensor are calculated dynamically based on the observed range of activations. This maximizes the accuracy of the quantization process as the activations make full use of the 8 quantized bits. In fully-quantized models, these parameters are fixed during model conversion, based on the range of the activation values observed using a representative dataset. The second difference between full quantization and dynamic range quantization is that the output of the Fully