# Half-precision Inference Doubles On-Device Inference Performance

DevFeed: [Half-precision Inference Doubles On-Device Inference Performance](<https://devfeed.tech/articles/half-precision-inference-doubles-on-device-inference-performance-7396.md>)

Original publisher: [Read original article](<https://blog.tensorflow.org/2023/11/half-precision-inference-doubles-on-device-inference-performance.html>)

Author: TensorFlow Blog (noreply@blogger.com)

Published: 2023-11-29T18:00:00Z

Content type: release

Language: en

Sources: [The TensorFlow Blog](<https://devfeed.tech/sources/the-tensorflow-blog.md>)

Topics: [Inference Performance](<https://devfeed.tech/topics/inference-performance.md>), [On-device AI](<https://devfeed.tech/topics/on-device-ai.md>), [LiteRT](<https://devfeed.tech/topics/litert.md>), [cpu](<https://devfeed.tech/topics/cpu.md>), [AI, ML & Data Engineering](<https://devfeed.tech/topics/ai-ml-data-engineering.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [announcement](<https://devfeed.tech/tags/announcement.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [inference](<https://devfeed.tech/tags/inference.md>), [inference-performance](<https://devfeed.tech/tags/inference-performance.md>), [learn](<https://devfeed.tech/tags/learn.md>), [on-device](<https://devfeed.tech/tags/on-device.md>), [tensorflow-lite](<https://devfeed.tech/tags/tensorflow-lite.md>)

## AI overview

TensorFlow Lite and XNNPack add generally available FP16 inference for supported ARM CPUs, reporting close to 2x speedups for floating-point models in production and benchmarked mobile and laptop environments.

## Source excerpt

Posted by Marat Dukhan and Frank Barchard, Software Engineers CPUs deliver the widest reach for ML inference and remain the default target for TensorFlow Lite. Consequently, improving CPU inference performance is a top priority, and we are excited to announce that we doubled floating-point inference performance in TensorFlow Lite's XNNPack backend by enabling half-precision inference on ARM CPUs. This means that more AI powered features may be deployed to older and lower tier devices. Traditionally, TensorFlow Lite supported two kinds of numerical computations in machine learning models: a) floating-point using IEEE 754 single-precision (32-bit) format and b) quantized using low-precision integers. While single-precision floating-point numbers provide maximum flexibility and ease of use, they come at the cost of 4X overhead in storage and memory and exhibit a performance overhead compared to 8-bit integer computations. In contrast, half-precision (FP16) floating-point numbers pose an interesting alternative balancing ease-of-use and performance: the processor needs to transfer twice fewer bytes and each vector operation produces twice more elements. By virtue of this property, FP16 inference paves the way for 2X speedup for floating-point models compared to the traditional FP32 way. For a long time FP16 inference on CPUs primarily remained a research topic, as the lack of hardware support for FP16 computations limited production use-cases. However, around 2017 new mobile chipsets started to include support for native FP16 computations, and by now most mobile phones, both on the high-end and the low-end. Building upon this broad availability, we are pleased to announce the general availability for half-precision inference in TensorFlow Lite and XNNPack. Performance Improvements Half-precision inference has already been battle-tested in production across Google Assistant, Google Meet, YouTube, and ML Kit, and demonstrated close to 2X speedups across a wide range of ne