# LiteRT

Google's on-device runtime for deploying high-performance machine-learning and generative-AI models on edge platforms.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Building Offline AI on Android: 5 LiteRT-LM Pitfalls and Fixes

DevFeed: [Building Offline AI on Android: 5 LiteRT-LM Pitfalls and Fixes](<https://devfeed.tech/articles/building-offline-ai-on-android-5-litert-lm-pitfalls-and-fixes-142887.md>)

Original publisher: [Read original article](<https://hackernoon.com/building-offline-ai-on-android-5-litert-lm-pitfalls-and-fixes?source=rss>)

Author: HandyDude

Published: 2026-11-19T08:23:24Z

Content type: article

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [Android](<https://devfeed.tech/topics/android.md>), [LiteRT](<https://devfeed.tech/topics/litert.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [Local-First](<https://devfeed.tech/topics/local-first.md>), [bugfixes](<https://devfeed.tech/topics/bugfixes.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [android](<https://devfeed.tech/tags/android.md>), [app](<https://devfeed.tech/tags/app.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [chat](<https://devfeed.tech/tags/chat.md>), [google-litert-lm](<https://devfeed.tech/tags/google-litert-lm.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [litert-lm](<https://devfeed.tech/tags/litert-lm.md>), [litert-lm-android](<https://devfeed.tech/tags/litert-lm-android.md>), [llm](<https://devfeed.tech/tags/llm.md>), [offline](<https://devfeed.tech/tags/offline.md>), [offline-first](<https://devfeed.tech/tags/offline-first.md>), [offline-llm](<https://devfeed.tech/tags/offline-llm.md>), [on-device-ai](<https://devfeed.tech/tags/on-device-ai.md>), [pitfalls](<https://devfeed.tech/tags/pitfalls.md>), [production](<https://devfeed.tech/tags/production.md>)

### AI overview

The account identifies five production pitfalls in an offline Android chat app using LiteRT-LM. GPU delegate memory use in versions ≤0.17 reached about 11 GB for a 0.8B int4 model and triggered foreground kills, so CPU is the default, with GPU opt-in and CPU fallback on initialization or inference failure. Optional `uses-native-library` declarations address Android 12+ OpenCL loading failures on some Samsung devices. Chat templates must parse LiteRT-LM's multipart content and sanitize serialized text before storage to avoid corrupting later turns; the issue can surface on turn 3 or later. `.litertlm` bundles may fail across runtime versions, so the author recommends tracking a minimum runtime and regenerating bundles on SDK upgrades. Since `Engine` and `Conversation` are not thread-safe, all native calls are serialized, and in-flight operations prevent mid-stream unloads. Reported warm initialization ranged from about 40 seconds for 0.8B on a Snapdragon S20-class device to 98 seconds on a Nord N200, and 168 seconds for 4B; the native container load accounted for most initialization time.

### Source excerpt

Production lessons from shipping an offline-first LLM chat app on Android with Google LiteRT-LM: GPU delegate OOMs, dlopen failures, thermal throttling and their fixes.

## On-Edge Acoustic Continual Anomaly Detection: A Cross-Platform Comparative Study

DevFeed: [On-Edge Acoustic Continual Anomaly Detection: A Cross-Platform Comparative Study](<https://devfeed.tech/articles/on-edge-acoustic-continual-anomaly-detection-a-cross-platform-comparative-study-132997.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.15465>)

Author: Rami Zewail, Yehia Ehab, Karim Ali, Ahmed Mahrous, Mohamed Awad, Bassem Mokhtar

Published: 2026-11-13T05:00:00Z

Content type: comparison

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LiteRT](<https://devfeed.tech/topics/litert.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Tensorflow](<https://devfeed.tech/topics/tensorflow.md>), [Embeddings](<https://devfeed.tech/topics/embeddings.md>), [Arduino](<https://devfeed.tech/topics/arduino.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [anomaly-detection](<https://devfeed.tech/tags/anomaly-detection.md>), [approach](<https://devfeed.tech/tags/approach.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [arduino](<https://devfeed.tech/tags/arduino.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [challenge](<https://devfeed.tech/tags/challenge.md>), [classification](<https://devfeed.tech/tags/classification.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [collection](<https://devfeed.tech/tags/collection.md>), [condition](<https://devfeed.tech/tags/condition.md>), [condition-monitoring](<https://devfeed.tech/tags/condition-monitoring.md>), [cross](<https://devfeed.tech/tags/cross.md>), [cross-platform](<https://devfeed.tech/tags/cross-platform.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-sd](<https://devfeed.tech/tags/cs-sd.md>), [data](<https://devfeed.tech/tags/data.md>), [detection](<https://devfeed.tech/tags/detection.md>), [edge](<https://devfeed.tech/tags/edge.md>), [platform](<https://devfeed.tech/tags/platform.md>), [study](<https://devfeed.tech/tags/study.md>)

### AI overview

The study reports comparable results for continual acoustic anomaly detection on two embedded platforms in a controlled speaker-replay evaluation using DCASE Task 2 machine sounds. Both use the same lightweight ShallowNet encoder and contrastive prototype learning model, with a 32-dimensional spectrogram embedding and on-device prototype updates via exponential moving average. The ESP32-S3 pairs with an external I2S digital microphone; the Arduino Nicla Vision uses an onboard PDM microphone. Both run the same TensorFlow Lite for Microcontrollers encoder, while Firebase and MQTT support remote data collection and retraining. Across five machine types, accuracy ranged from 82.35% to 95.00% on the ESP32-S3 and 79.41% to 95.00% on the Nicla Vision. The reported comparison suggests the architecture transfers across these hardware and sensor configurations; the evidence here is limited to the stated controlled protocol.

### Source excerpt

arXiv:2611.15465v1 Announce Type: cross Abstract: Acoustic Anomaly Detection (AAD) serves as a non-contact approach for machine condition monitoring, yet much of the existing research is evaluated offline and rarely addresses embedded hardware constraints or microphone variations under operational drift. This paper presents a comparative evaluation of on-edge continual AAD across two embedded platforms the ESP32-S3 (with an external I2S digital microphone) and the Arduino Nicla Vision (with an onboard Pulse Density Modulation (PDM) microphone) using a shared detection model. The model utilizes a lightweight ShallowNet encoder and contrastive prototype learning to map spectrograms into a 32-dimensional embedding space. Both boards run an identical TensorFlow Lite for Microcontrollers (TFLM) encoder, performing on-device prototype learning updated via an exponential moving average. A cloud pipeline orchestrated through Firebase and MQTT enables remote data collection and retraining. Evaluated via a controlled speaker-replay protocol using standardized Detection and Classification of Acoustic Scenes and Events (DCASE) Challenge Task 2 machine sounds, the ESP32-S3 achieved 82.35% to 95.00% accuracy across five machine types, while the Nicla Vision achieved 79.41% to 95.00%. The two platforms demonstrated comparable performance, confirming that the lightweight prototypical architecture transfers reliably across distinct hardware and sensor tiers.

## Google Launches LiteRT Tensor API for Building, Customizing, and Optimizing ML Models for On-Device

DevFeed: [Google Launches LiteRT Tensor API for Building, Customizing, and Optimizing ML Models for On-Device](<https://devfeed.tech/articles/google-launches-litert-tensor-api-for-building-customizing-and-optimizing-ml-models-for-on-device-115569.md>)

Original publisher: [Read original article](<https://www.infoq.com/news/2026/11/litert-tensor-api/>)

Author: Sergio De Simone

Published: 2026-11-03T19:00:00Z

Content type: news

Language: en

Sources: [InfoQ](<https://devfeed.tech/sources/infoq.md>)

Topics: [LiteRT](<https://devfeed.tech/topics/litert.md>), [Model Development](<https://devfeed.tech/topics/model-development.md>), [hugging face](<https://devfeed.tech/topics/hugging-face.md>), [OpenAI](<https://devfeed.tech/topics/openai.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-coding-agents](<https://devfeed.tech/tags/ai-coding-agents.md>), [ai-ml-data-engineering](<https://devfeed.tech/tags/ai-ml-data-engineering.md>), [android](<https://devfeed.tech/tags/android.md>), [api](<https://devfeed.tech/tags/api.md>), [authoring](<https://devfeed.tech/tags/authoring.md>), [build](<https://devfeed.tech/tags/build.md>), [building](<https://devfeed.tech/tags/building.md>), [c-plus-plus](<https://devfeed.tech/tags/c-plus-plus.md>), [cross-platform](<https://devfeed.tech/tags/cross-platform.md>), [developers](<https://devfeed.tech/tags/developers.md>), [development](<https://devfeed.tech/tags/development.md>), [edge-computing](<https://devfeed.tech/tags/edge-computing.md>), [execution](<https://devfeed.tech/tags/execution.md>), [flexibility](<https://devfeed.tech/tags/flexibility.md>), [google](<https://devfeed.tech/tags/google.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [introduced](<https://devfeed.tech/tags/introduced.md>), [ios](<https://devfeed.tech/tags/ios.md>), [launches](<https://devfeed.tech/tags/launches.md>), [learning](<https://devfeed.tech/tags/learning.md>), [litert](<https://devfeed.tech/tags/litert.md>), [litert-tensor-api](<https://devfeed.tech/tags/litert-tensor-api.md>), [ml](<https://devfeed.tech/tags/ml.md>), [ml-data-engineering](<https://devfeed.tech/tags/ml-data-engineering.md>), [ml-models](<https://devfeed.tech/tags/ml-models.md>), [mobile](<https://devfeed.tech/tags/mobile.md>), [models](<https://devfeed.tech/tags/models.md>), [news](<https://devfeed.tech/tags/news.md>), [on-device](<https://devfeed.tech/tags/on-device.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [optimizing](<https://devfeed.tech/tags/optimizing.md>), [tensorflow](<https://devfeed.tech/tags/tensorflow.md>)

### AI overview

Google's LiteRT Tensor API lets developers define machine-learning computation graphs operation by operation in C++, then run the same graphs through LiteRT on CPU, GPU, and NPU targets without rewriting for each backend. The source highlights direct loading of checkpoint weights, including `.safetensors`, and zero-copy buffer interoperability across preprocessing, inference, and postprocessing. It says coding agents can generate and optimize the C++ graphs. In Google's reported SAM 2 tests, the same code ran on GPUs using Metal on a Mac and iPhone and OpenCL on Android; on a MacBook Pro, it was about 1.5x faster than Apple MLX in the stated fp16 image-encoder plus mask-decoder comparison. Arm's Laurence Moroney advised starting with model conversion and using the C++ API when graph-level control or accelerator-side preprocessing and postprocessing are needed. Planned improvements include ML Drift support, more NPU-friendly operations, full-graph quantization, and broader dynamic-shape support.

### Source excerpt

Google has introduced the LiteRT Tensor API, a lightweight C++ API that allows developers to build, customize, and optimize machine learning models directly for on-device execution. According to Google, it combines the flexibility of native model authoring with cross-platform portability. By Sergio De Simone

## Smaller apps and faster GPU inference with LiteRT in Google Play services

DevFeed: [Smaller apps and faster GPU inference with LiteRT in Google Play services](<https://devfeed.tech/articles/smaller-apps-and-faster-gpu-inference-with-litert-in-google-play-services-105743.md>)

Original publisher: [Read original article](<https://developers.googleblog.com/smaller-apps-and-faster-gpu-inference-with-litert-in-google-play-services/>)

Author: Google AI Edge Team; Android ML Team

Published: 2026-10-29T03:38:31.321833Z

Content type: release

Language: en

Sources: [Google Developers Blog](<https://devfeed.tech/sources/google-developers-blog.md>)

Topics: [LiteRT](<https://devfeed.tech/topics/litert.md>)

Tags: [about](<https://devfeed.tech/tags/about.md>), [android](<https://devfeed.tech/tags/android.md>), [android-studio](<https://devfeed.tech/tags/android-studio.md>), [api](<https://devfeed.tech/tags/api.md>), [app](<https://devfeed.tech/tags/app.md>), [apps](<https://devfeed.tech/tags/apps.md>), [can](<https://devfeed.tech/tags/can.md>), [dependencies](<https://devfeed.tech/tags/dependencies.md>), [developers](<https://devfeed.tech/tags/developers.md>), [faster](<https://devfeed.tech/tags/faster.md>), [google](<https://devfeed.tech/tags/google.md>), [google-play](<https://devfeed.tech/tags/google-play.md>), [google-play-services](<https://devfeed.tech/tags/google-play-services.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpu-inference](<https://devfeed.tech/tags/gpu-inference.md>), [inference](<https://devfeed.tech/tags/inference.md>), [litert](<https://devfeed.tech/tags/litert.md>), [play](<https://devfeed.tech/tags/play.md>), [services](<https://devfeed.tech/tags/services.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

LiteRT in Google Play services adds GPU inference through the CompiledModel API, which the source says ran about 1.4x faster than the legacy GPU delegate in its testing. Moving from bundled LiteRT can reduce app size by about 8 MB because Play services supplies the runtime and GPU accelerator. Developers can add `com.google.android.gms:play-services-litert:17.0.0` or later, remove the bundled `litert` dependency, initialize with GPU enabled, and pass the environment to CompiledModel. Existing Interpreter code continues to work on CPU; GPU use requires moving to CompiledModel. The Android Studio LiteRT skill can automate migration, but the source recommends reviewing its changes, building, and comparing outputs on a real device. Legacy GPU and custom delegate APIs are deprecated, NNAPI removal begins in 2027, and deprecated API code will not compile against the latest SDK.

### Source excerpt

LiteRT in Google Play services now offers GPU inference powered by ML Drift, delivering about 1.4x faster performance alongside the new CompiledModel API. Switching from bundled binaries reduces app size by approximately 8 MB while enabling seamless, automatic runtime updates across billions of devices. Developers can easily adopt these updates by updating their dependencies and utilizing the new Android Studio LiteRT skill for automated migration.

## Google ML Drift Accelerates On-Device AI Inference Across Diverse GPUs

DevFeed: [Google ML Drift Accelerates On-Device AI Inference Across Diverse GPUs](<https://devfeed.tech/articles/google-ml-drift-accelerates-on-device-ai-inference-across-diverse-gpus-99272.md>)

Original publisher: [Read original article](<https://www.infoq.com/news/2026/10/google-ml-drift/>)

Author: Sergio De Simone

Published: 2026-10-25T17:00:00Z

Content type: news

Language: en

Sources: [InfoQ](<https://devfeed.tech/sources/infoq.md>)

Topics: [On-device AI](<https://devfeed.tech/topics/on-device-ai.md>), [LiteRT](<https://devfeed.tech/topics/litert.md>), [OpenCL](<https://devfeed.tech/topics/opencl.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [gemma](<https://devfeed.tech/topics/gemma.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-and-machine-learning](<https://devfeed.tech/tags/ai-and-machine-learning.md>), [ai-inference](<https://devfeed.tech/tags/ai-inference.md>), [ai-ml-data-engineering](<https://devfeed.tech/tags/ai-ml-data-engineering.md>), [apache-2-0](<https://devfeed.tech/tags/apache-2-0.md>), [apis](<https://devfeed.tech/tags/apis.md>), [compute](<https://devfeed.tech/tags/compute.md>), [cross-platform](<https://devfeed.tech/tags/cross-platform.md>), [development](<https://devfeed.tech/tags/development.md>), [device](<https://devfeed.tech/tags/device.md>), [edge](<https://devfeed.tech/tags/edge.md>), [engine](<https://devfeed.tech/tags/engine.md>), [google](<https://devfeed.tech/tags/google.md>), [google-ml-drift](<https://devfeed.tech/tags/google-ml-drift.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpu-compute](<https://devfeed.tech/tags/gpu-compute.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [inference](<https://devfeed.tech/tags/inference.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [local-inference](<https://devfeed.tech/tags/local-inference.md>), [low-level](<https://devfeed.tech/tags/low-level.md>), [ml](<https://devfeed.tech/tags/ml.md>), [ml-data-engineering](<https://devfeed.tech/tags/ml-data-engineering.md>), [mobile](<https://devfeed.tech/tags/mobile.md>), [news](<https://devfeed.tech/tags/news.md>), [on-device](<https://devfeed.tech/tags/on-device.md>), [on-device-ai](<https://devfeed.tech/tags/on-device-ai.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [tensorflow](<https://devfeed.tech/tags/tensorflow.md>)

### AI overview

Google's open-source ML Drift is a cross-platform GPU engine for on-device inference and the acceleration core of LiteRT. It abstracts GPU hardware and APIs, supporting OpenGL ES, OpenCL, Metal, and WebGPU; desktop inference uses Chromium's Dawn on Windows and Linux, while macOS retains native Metal support. Compared with the legacy TensorFlow Lite GPU delegate, ML Drift adds unified shaders through tensor virtualization, custom operations, five-dimensional tensors, and optimizations for Arm, Intel, and Qualcomm GPUs. Google reports 12% lower memory overhead than llama.cpp and MLX LM in workstation Gemma benchmarks, plus app-specific latency or performance gains; these are company-reported results, and the source says Google provided no comparison with Core ML on recent iPhones. Google will stop updating the legacy delegate and recommends migrating existing models to the LiteRT ML Drift GPU accelerator. Developers can use that accelerator or the lower-level C++ API to build custom compute graphs and shaders.

### Source excerpt

Google has released ML Drift, an open-source, cross-platform GPU compute engine designed for on-device AI and machine learning inference. Licensed under Apache 2.0, it abstracts the complexities of different GPU hardware and low-level APIs, supporting OpenGL ES, OpenCL, Metal, and WebGPU. By Sergio De Simone

## Bringing Visual Graph Debugging to Custom Edge Compilers with Model Explorer

DevFeed: [Bringing Visual Graph Debugging to Custom Edge Compilers with Model Explorer](<https://devfeed.tech/articles/bringing-visual-graph-debugging-to-custom-edge-compilers-with-model-explorer-91264.md>)

Original publisher: [Read original article](<https://developers.googleblog.com/bringing-visual-graph-debugging-to-custom-edge-compilers-with-model-explorer/>)

Author: Eric Yang; Alan Ma

Published: 2026-10-21T02:36:15.506968Z

Content type: article

Language: en

Sources: [Google Developers Blog](<https://devfeed.tech/sources/google-developers-blog.md>)

Topics: [model-deployment](<https://devfeed.tech/topics/model-deployment.md>), [LiteRT](<https://devfeed.tech/topics/litert.md>), [Model Optimizer](<https://devfeed.tech/topics/model-optimizer.md>), [Compiler](<https://devfeed.tech/topics/compiler.md>)

Tags: [adapters](<https://devfeed.tech/tags/adapters.md>), [ai](<https://devfeed.tech/tags/ai.md>), [bottlenecks](<https://devfeed.tech/tags/bottlenecks.md>), [can](<https://devfeed.tech/tags/can.md>), [compiler](<https://devfeed.tech/tags/compiler.md>), [custom](<https://devfeed.tech/tags/custom.md>), [delegation](<https://devfeed.tech/tags/delegation.md>), [developers](<https://devfeed.tech/tags/developers.md>), [device](<https://devfeed.tech/tags/device.md>), [edge](<https://devfeed.tech/tags/edge.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [execution](<https://devfeed.tech/tags/execution.md>), [explorer](<https://devfeed.tech/tags/explorer.md>), [google](<https://devfeed.tech/tags/google.md>), [google-ai](<https://devfeed.tech/tags/google-ai.md>), [graph](<https://devfeed.tech/tags/graph.md>), [graph-visualization](<https://devfeed.tech/tags/graph-visualization.md>), [graphs](<https://devfeed.tech/tags/graphs.md>), [model](<https://devfeed.tech/tags/model.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

Arm's open-source adapters extend Google AI Edge Model Explorer to visualize TOSA, VGF, and PTE representations and profiling data for on-device model debugging. In an ExecuTorch Arm backend VGF export, inspecting graph nodes helped identify unintended TRANSPOSE operators introduced by a backend change. For a YOLO model, split-pane CPU and Ethos-U85 NPU graphs showed delegation as a single EthosUBackend block and could expose operators left un-delegated in partially supported models. On an Ethos-U FVP simulation, a node-data plugin overlaid cycle-accurate traces as heatmaps, highlighting a TRANSPOSE node as the main latency bottleneck. The excerpt describes local graph inspection and profiling, but does not provide setup details.

### Source excerpt

Google AI Edge Model Explorer now supports custom compiler intermediate representations and open-source adapters like TOSA, VGF, and PTE for on-device machine learning. Developers can leverage hierarchical graph visualization, split-pane views, and per-node trace overlays to catch export regressions and verify hardware delegation. These tools enable engineering teams to transparently debug execution graphs and pinpoint cycle-level latency bottlenecks locally.

## Native Model Authoring for Edge with LiteRT Tensor API

DevFeed: [Native Model Authoring for Edge with LiteRT Tensor API](<https://devfeed.tech/articles/native-model-authoring-for-edge-with-litert-tensor-api-90416.md>)

Original publisher: [Read original article](<https://developers.googleblog.com/native-model-authoring-for-edge-with-litert-tensor-api/>)

Author: Ping Yu; Quentin Khan; Chintan Parikh; Lu Wang

Published: 2026-10-20T02:35:00.194306Z

Content type: release

Language: en

Sources: [Google Developers Blog](<https://devfeed.tech/sources/google-developers-blog.md>)

Topics: [LiteRT](<https://devfeed.tech/topics/litert.md>), [Advanced AI workflows](<https://devfeed.tech/topics/advanced-ai-workflows.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [PyTorch](<https://devfeed.tech/topics/pytorch.md>)

Tags: [antigravity](<https://devfeed.tech/tags/antigravity.md>), [c-plus-plus](<https://devfeed.tech/tags/c-plus-plus.md>), [coding](<https://devfeed.tech/tags/coding.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [litert](<https://devfeed.tech/tags/litert.md>), [ml](<https://devfeed.tech/tags/ml.md>)

### AI overview

Google introduces the LiteRT Tensor API, a C++ layer for authoring machine-learning computation graphs that can run across CPU, GPU, and NPU backends. The article presents a Segment Anything Model 2 demo and describes how shared accelerator buffers and graph-level changes can reduce data transfers and improve frame rates. It also discusses the API's potential use in AI coding-agent workflows.

### Source excerpt

Meet the LiteRT Tensor API: giving developers native C++ control over the model graph to unify end-to-end ML computations on heterogeneous edge hardware.

## ML Drift: Next-Gen GPU AI/ML Inference at the Edge

DevFeed: [ML Drift: Next-Gen GPU AI/ML Inference at the Edge](<https://devfeed.tech/articles/ml-drift-next-gen-gpu-ai-ml-inference-at-the-edge-77465.md>)

Original publisher: [Read original article](<https://developers.googleblog.com/ml-drift-next-gen-gpu-aiml-inference-at-the-edge/>)

Author: Chintan Parikh; Juhyun Lee

Published: 2026-10-09T02:20:22.696109Z

Content type: release

Language: en

Sources: [Google Developers Blog](<https://devfeed.tech/sources/google-developers-blog.md>)

Topics: [LiteRT](<https://devfeed.tech/topics/litert.md>), [On-device AI](<https://devfeed.tech/topics/on-device-ai.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [OpenGL](<https://devfeed.tech/topics/opengl.md>), [Warp](<https://devfeed.tech/topics/warp.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-ml](<https://devfeed.tech/tags/ai-ml.md>), [developers](<https://devfeed.tech/tags/developers.md>), [device](<https://devfeed.tech/tags/device.md>), [edge](<https://devfeed.tech/tags/edge.md>), [effects](<https://devfeed.tech/tags/effects.md>), [generative-ai](<https://devfeed.tech/tags/generative-ai.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpu-compute](<https://devfeed.tech/tags/gpu-compute.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [high-performance](<https://devfeed.tech/tags/high-performance.md>), [inference](<https://devfeed.tech/tags/inference.md>), [ml](<https://devfeed.tech/tags/ml.md>), [mobile](<https://devfeed.tech/tags/mobile.md>), [on-device](<https://devfeed.tech/tags/on-device.md>), [on-device-ai](<https://devfeed.tech/tags/on-device-ai.md>), [opencl](<https://devfeed.tech/tags/opencl.md>), [opengl](<https://devfeed.tech/tags/opengl.md>), [performance](<https://devfeed.tech/tags/performance.md>), [webgpu](<https://devfeed.tech/tags/webgpu.md>)

### AI overview

Google AI Edge announced ML Drift, an open-source GPU compute engine for on-device AI and machine learning inference. It abstracts GPU hardware and APIs across platforms, supports workloads from computer vision to generative AI, and serves as the GPU acceleration engine in LiteRT while also being available as a standalone library. The announcement describes shader and tensor support changes, optimizations for edge LLM inference, and desktop previews.

### Source excerpt

Google AI Edge has introduced ML Drift, a high-performance, universal GPU compute framework designed to accelerate on-device AI and machine learning inference. By abstracting the complexities of hardware and low-level APIs--including OpenGL ES, OpenCL, Metal, and WebGPU--ML Drift enables developers to build real-time, interactive ML experiences, from sophisticated video effects to generative AI, across mobile and desktop platforms.

## Bring multimodal semantic search to the edge with EmbeddingGemma 2

DevFeed: [Bring multimodal semantic search to the edge with EmbeddingGemma 2](<https://devfeed.tech/articles/bring-multimodal-semantic-search-to-the-edge-with-embeddinggemma-2-65872.md>)

Original publisher: [Read original article](<https://developers.googleblog.com/google-ai-edge-with-embeddinggemma-2/>)

Author: Google AI Edge Team; Android ML Team

Published: 2026-10-07T02:19:03.028931Z

Content type: release

Language: en

Sources: [Google Developers Blog](<https://devfeed.tech/sources/google-developers-blog.md>)

Topics: [Embeddings](<https://devfeed.tech/topics/embeddings.md>), [LiteRT](<https://devfeed.tech/topics/litert.md>), [Real-time AI interactions](<https://devfeed.tech/topics/real-time-ai-interactions.md>)

Tags: [cpu](<https://devfeed.tech/tags/cpu.md>), [edge](<https://devfeed.tech/tags/edge.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [litert](<https://devfeed.tech/tags/litert.md>), [low-latency](<https://devfeed.tech/tags/low-latency.md>), [mediapipe](<https://devfeed.tech/tags/mediapipe.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [npu](<https://devfeed.tech/tags/npu.md>), [on-device](<https://devfeed.tech/tags/on-device.md>), [search](<https://devfeed.tech/tags/search.md>), [semantic-search](<https://devfeed.tech/tags/semantic-search.md>)

### AI overview

Google introduces EmbeddingGemma 2, a compact open-weight multimodal model that embeds text, images, video, and audio in a shared vector space for local retrieval. The post presents on-device media search and video moment finding, and describes developer paths through MediaPipe Tasks and LiteRT for cross-platform deployment and hardware acceleration.

### Source excerpt

EmbeddingGemma 2 is a new 740M open-weight multimodal model that maps text, images, video, and audio into a unified vector space for privacy-first, on-device retrieval. Developers can easily integrate these capabilities cross-platform using MediaPipe Tasks or optimize fine-grained performance across CPU, GPU, and NPU accelerators with LiteRT. The model enables ultra-low-latency local solutions like search-as-you-type media retrieval, keyframe video moments finding, and zero-shot intent routing.

## Fine-Tuning Small Models for On-Device Agents: Train, Convert, and Measure What Changed

DevFeed: [Fine-Tuning Small Models for On-Device Agents: Train, Convert, and Measure What Changed](<https://devfeed.tech/articles/fine-tuning-small-models-for-on-device-agents-train-convert-and-measure-what-changed-65297.md>)

Original publisher: [Read original article](<https://medium.com/google-developer-experts/fine-tuning-small-models-for-on-device-agents-train-convert-and-measure-what-changed-f7f494a044f3?source=rss----a67bd6fa7d58---4>)

Author: Sasha Denisov

Published: 2026-10-06T07:20:54Z

Content type: tutorial

Language: en

Sources: [Google Developer Experts](<https://devfeed.tech/sources/google-developer-experts-medium.md>)

Topics: [peft](<https://devfeed.tech/topics/peft.md>), [lora](<https://devfeed.tech/topics/lora.md>), [LiteRT](<https://devfeed.tech/topics/litert.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [ML model versioning](<https://devfeed.tech/topics/ml-model-versioning.md>), [Deployment Strategies](<https://devfeed.tech/topics/deployment-strategies.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [ai-on-device](<https://devfeed.tech/tags/ai-on-device.md>), [apache](<https://devfeed.tech/tags/apache.md>), [build](<https://devfeed.tech/tags/build.md>), [data](<https://devfeed.tech/tags/data.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [edge](<https://devfeed.tech/tags/edge.md>), [edge-ai](<https://devfeed.tech/tags/edge-ai.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [hugging-face](<https://devfeed.tech/tags/hugging-face.md>), [inference](<https://devfeed.tech/tags/inference.md>), [litert](<https://devfeed.tech/tags/litert.md>), [lora](<https://devfeed.tech/tags/lora.md>), [mediapipe](<https://devfeed.tech/tags/mediapipe.md>), [model](<https://devfeed.tech/tags/model.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [trl](<https://devfeed.tech/tags/trl.md>), [tutorial](<https://devfeed.tech/tags/tutorial.md>)

### AI overview

The article presents litetune, an open-source command-line workflow for fine-tuning small models with LoRA, converting them to LiteRT-LM, and verifying converted artifacts against float models on held-out data. In the reported tests, eight-bit conversions stayed close to float on the measured workloads, while four-bit results varied by model and recipe and could show substantial quality losses. The author emphasizes testing the converted artifact and measuring performance on its target device.

### Source excerpt

How litetune rebuilds the LoRA-to-device workflow and measures the model that actually ships. In February 2025, I published a ten-step guide to fine-tuning Gemma with LoRA in Colab, converting the model, and running it on a phone. I build Flutter Edge AI (formerly flutter_gemma), a plugin for running these models directly on the device, so fine-tuning was the natural next step. Today, following that guide step by step no longer works. I first tried to update it, but replacing one stage led to another. LoRA was the only part that stayed the same. So this is not a patched version of the old tutorial. It is a new workflow, with one stage the original did not have: verifying the converted model on held-out data. What Changed in the Workflow The old guide was built around the first Gemma models -- 2B and 7B -- and MediaPipe for on-device inference. The entire path from training to a MediaPipe-ready artifact fit inside a single Colab notebook. Today, its parts are spread across Hugging Face, TRL, litert-converter, and LiteRT-LM. They have different dependencies, and export settings vary by model architecture. LoRA training itself still looks familiar -- the complexity now sits around it. Before training, you need to validate the data and create and record a held-out split. After training, you need to merge the adapter, select the export settings, build a .litertlm file, run it through LiteRT-LM, and compare its outputs with the original model. A working checkpoint does not guarantee that the final artifact will behave the same way. When it became clear that the old notebook could not simply be updated, I decided to put the modern workflow into one tool. That is how litetune started: an open-source command-line tool, released under Apache 2.0, that takes a dataset all the way to a verified .litertlm. Its job is to remove the handwritten glue code between a dataset and a model that an application can run. Instead of maintaining separate scripts for data preparation, LoRA train

## Building On-Device AI Agents on Android with Google ADK -- Part 2: Giving Your Agent Hands --...

DevFeed: [Building On-Device AI Agents on Android with Google ADK -- Part 2: Giving Your Agent Hands --...](<https://devfeed.tech/articles/building-on-device-ai-agents-on-android-with-google-adk-part-2-giving-your-agent-hands-61528.md>)

Original publisher: [Read original article](<https://medium.com/google-developer-experts/building-on-device-ai-agents-on-android-with-google-adk-part-2-giving-your-agent-hands-850308e17154?source=rss----a67bd6fa7d58---4>)

Author: Pankaj Rai

Published: 2026-09-29T01:50:37Z

Content type: tutorial

Language: en

Sources: [Google Developer Experts](<https://devfeed.tech/sources/google-developer-experts-medium.md>)

Topics: [google-adk](<https://devfeed.tech/topics/google-adk.md>), [On-device AI](<https://devfeed.tech/topics/on-device-ai.md>), [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [Android](<https://devfeed.tech/topics/android.md>), [LiteRT](<https://devfeed.tech/topics/litert.md>), [gemma](<https://devfeed.tech/topics/gemma.md>)

Tags: [adk](<https://devfeed.tech/tags/adk.md>), [agent](<https://devfeed.tech/tags/agent.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [android](<https://devfeed.tech/tags/android.md>), [android-app-development](<https://devfeed.tech/tags/android-app-development.md>), [androiddev](<https://devfeed.tech/tags/androiddev.md>), [apis](<https://devfeed.tech/tags/apis.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [call](<https://devfeed.tech/tags/call.md>), [compile](<https://devfeed.tech/tags/compile.md>), [custom](<https://devfeed.tech/tags/custom.md>), [functions](<https://devfeed.tech/tags/functions.md>), [google](<https://devfeed.tech/tags/google.md>), [google-adk](<https://devfeed.tech/tags/google-adk.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [inspect](<https://devfeed.tech/tags/inspect.md>), [ksp](<https://devfeed.tech/tags/ksp.md>), [on-device](<https://devfeed.tech/tags/on-device.md>), [on-device-ai](<https://devfeed.tech/tags/on-device-ai.md>), [time](<https://devfeed.tech/tags/time.md>), [tool](<https://devfeed.tech/tags/tool.md>)

### AI overview

An on-device Gemma model loaded with LiteRT-LM cannot answer questions about current phone state from training knowledge alone. Google ADK tools provide executable capabilities: the model emits a structured function call naming a tool and its arguments, waits for that tool to run, and can then use the result. The excerpt introduces BaseTool and compile-time @Tool KSP bindings as ways to add capabilities, but does not provide enough detail to compare them.

### Source excerpt

Empower your on-device AI Agent to inspect hardware, call Android APIs, and run custom functions using BaseTool and compile-time @Tool KSP... Continue reading on Google Developer Experts "

## Introducing Support for Local AI Models in the Antigravity SDK

DevFeed: [Introducing Support for Local AI Models in the Antigravity SDK](<https://devfeed.tech/articles/introducing-support-for-local-ai-models-in-the-antigravity-sdk-58908.md>)

Original publisher: [Read original article](<https://developers.googleblog.com/introducing-support-for-local-ai-models-in-the-antigravity-sdk/>)

Author: Sachin Kotwani; Taylor Mullen

Published: 2026-09-24T01:36:29.465879Z

Content type: release

Language: en

Sources: [Google Developers Blog](<https://devfeed.tech/sources/google-developers-blog.md>)

Topics: [google-antigravity](<https://devfeed.tech/topics/google-antigravity.md>), [Local AI](<https://devfeed.tech/topics/local-ai.md>), [SDK](<https://devfeed.tech/topics/sdk.md>), [agentic workflows](<https://devfeed.tech/topics/agentic-workflows.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [AI Models](<https://devfeed.tech/topics/ai-models.md>), [gemma](<https://devfeed.tech/topics/gemma.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [LiteRT](<https://devfeed.tech/topics/litert.md>), [Google](<https://devfeed.tech/topics/google.md>), [Ollama](<https://devfeed.tech/topics/ollama.md>), [vllm](<https://devfeed.tech/topics/vllm.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-workflows](<https://devfeed.tech/tags/agentic-workflows.md>), [ai-models](<https://devfeed.tech/tags/ai-models.md>), [code](<https://devfeed.tech/tags/code.md>), [device](<https://devfeed.tech/tags/device.md>), [gemma](<https://devfeed.tech/tags/gemma.md>), [gemma-4](<https://devfeed.tech/tags/gemma-4.md>), [google](<https://devfeed.tech/tags/google.md>), [google-antigravity](<https://devfeed.tech/tags/google-antigravity.md>), [inference](<https://devfeed.tech/tags/inference.md>), [litert](<https://devfeed.tech/tags/litert.md>), [local-ai](<https://devfeed.tech/tags/local-ai.md>), [ollama](<https://devfeed.tech/tags/ollama.md>), [privacy](<https://devfeed.tech/tags/privacy.md>), [sdk](<https://devfeed.tech/tags/sdk.md>), [vllm](<https://devfeed.tech/tags/vllm.md>), [workflows](<https://devfeed.tech/tags/workflows.md>)

### AI overview

Google announces local AI model support in the Antigravity SDK, including offline agentic workflows with Gemma 4 26B A4B through LiteRT. The update also supports hybrid cloud-local orchestration and OpenAI-compatible servers such as Ollama and vLLM.

### Source excerpt

The Google Antigravity SDK now empowers developers to execute offline, agentic workflows locally using models like Gemma 4 26B A4B via LiteRT. This update facilitates powerful hybrid orchestration architectures, allowing a cloud model to act as a lightweight planner while local models securely handle token-intensive tasks--like code auditing and patching--directly on-device. Furthermore, the SDK provides drop-in support for OpenAI-compatible inference servers like Ollama and vLLM, enabling the seamless creation of privacy-first, autonomous local utilities.

## On-Device AI Series (Part 5): LiteRT-LM

DevFeed: [On-Device AI Series (Part 5): LiteRT-LM](<https://devfeed.tech/articles/on-device-ai-series-part-5-litert-lm-22949.md>)

Original publisher: [Read original article](<https://proandroiddev.com/on-device-ai-series-part-5-litert-lm-d6c23b102094?source=rss----c72404660798---4>)

Author: Oğuzhan Aslan

Published: 2026-09-14T05:59:12Z

Content type: tutorial

Language: en

Sources: [ProAndroidDev](<https://devfeed.tech/sources/proandroiddev-medium.md>)

Topics: [On-device AI](<https://devfeed.tech/topics/on-device-ai.md>), [LiteRT](<https://devfeed.tech/topics/litert.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [AI Context Management](<https://devfeed.tech/topics/ai-context-management.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [Agent Harness](<https://devfeed.tech/topics/agent-harness.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [airplane](<https://devfeed.tech/tags/airplane.md>), [all](<https://devfeed.tech/tags/all.md>), [android](<https://devfeed.tech/tags/android.md>), [android-development](<https://devfeed.tech/tags/android-development.md>), [api](<https://devfeed.tech/tags/api.md>), [api-key](<https://devfeed.tech/tags/api-key.md>), [app](<https://devfeed.tech/tags/app.md>), [avoid](<https://devfeed.tech/tags/avoid.md>), [backend](<https://devfeed.tech/tags/backend.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [bill](<https://devfeed.tech/tags/bill.md>), [blocking](<https://devfeed.tech/tags/blocking.md>), [bookkeeping](<https://devfeed.tech/tags/bookkeeping.md>), [box](<https://devfeed.tech/tags/box.md>), [budget](<https://devfeed.tech/tags/budget.md>), [build](<https://devfeed.tech/tags/build.md>), [device](<https://devfeed.tech/tags/device.md>), [litert](<https://devfeed.tech/tags/litert.md>), [on-device](<https://devfeed.tech/tags/on-device.md>), [on-device-ai](<https://devfeed.tech/tags/on-device-ai.md>), [programming](<https://devfeed.tech/tags/programming.md>), [series](<https://devfeed.tech/tags/series.md>), [software-development](<https://devfeed.tech/tags/software-development.md>)

### AI overview

LiteRT-LM supports offline, on-device LLM chat with streaming, session state and KV-cache management, and runtime CPU/GPU selection. An Engine loads and reuses the model; lightweight Sessions hold conversation state and can be replaced without reloading it. Developers must size maxNumTokens for combined input and output, configure sampling, and construct system prompts and tool calling using model-specific text formats. Multimodal input uses Conversation rather than Session, and the two APIs share a native slot. The excerpt also describes a Gradle fix for conflicting native libraries that can cause SIGSEGV. A reported Gemma E2B benchmark ran at 2.9 tokens/sec on CPU and 5.0 on GPU, with performance dependent on hardware. Tradeoffs include local privacy and no inference-time network or per-token bill, alongside lower capability than large cloud models and responsibility for latency, memory, and prompt handling.

### Source excerpt

Put your phone in airplane mode. Open the app, type a question, and watch the answer arrive one token at a time -- no spinner waiting on a network round-trip, no API key, no per-token bill, and nothing you typed ever leaving the device. LiteRT-LM removes the genuinely hard parts of running an LLM on-device -- KV-cache management, token streaming, backend selection -- but it doesn't remove your job so much as relocate it. What's left on your plate is a short, specific list: sizing a combined input+output token budget, owning your own sampling defaults, hand-building system prompts and tool calling out of raw text, and one native-library collision that presents as a SIGSEGV rather than a build error. Know those going in and the API itself is a clean three-step pattern. We'll get there in that order: Why you'd choose this runtime and what it costs you versus the cloud. The Engine/Session model you need to read the code at all. Real implementation samples -- streaming, system prompts and tool calling, multimodal inputs, thinking mode, and CPU-vs-GPU benchmarking. The anti-patterns to avoid. A developer-friendliness rating on the same rubric as Parts 1-4. Why Use LiteRT-LM? You reach for LiteRT-LM instead of hand-rolling generation on top of raw LiteRT when: You need multi-turn conversation, not single-shot inference -- session state and KV-cache bookkeeping are handled for you, and resetting a conversation is a session swap, not a model reload. You need streaming output -- token-by-token delivery for a responsive chat UI, instead of a blocking call that returns everything at once. You're choosing between CPU and GPU per device -- the explicit backend parameter turns that into a runtime decision instead of a build-time guess. You want a pre-converted model without doing your own PyTorch-to-LiteRT conversion work -- the Model Zoo covers Gemma, Qwen, Llama, and more out of the box. You're willing to own sampling -- the engine won't pick sane decoding defaults for you; that's on the

## Mastering Edge AI on Raspberry Pi with LiteRT and Gemma

DevFeed: [Mastering Edge AI on Raspberry Pi with LiteRT and Gemma](<https://devfeed.tech/articles/mastering-edge-ai-on-raspberry-pi-with-litert-and-gemma-4215.md>)

Original publisher: [Read original article](<https://developers.googleblog.com/mastering-edge-ai-on-raspberry-pi-with-litert-and-gemma/>)

Author: Lu Wang; Terry Heo; Naushir Patuck; José María Casanova

Published: 2026-09-12T11:04:33.891311Z

Content type: tutorial

Language: en

Sources: [Google Developers Blog](<https://devfeed.tech/sources/google-developers-blog.md>)

Topics: [LiteRT](<https://devfeed.tech/topics/litert.md>), [On-device AI](<https://devfeed.tech/topics/on-device-ai.md>), [Local AI](<https://devfeed.tech/topics/local-ai.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Google AI](<https://devfeed.tech/topics/google-ai.md>), [cpu](<https://devfeed.tech/topics/cpu.md>), [Security, Privacy and Abuse Prevention](<https://devfeed.tech/topics/security-privacy-and-abuse-prevention.md>)

Tags: [cli](<https://devfeed.tech/tags/cli.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [edge](<https://devfeed.tech/tags/edge.md>), [gemma](<https://devfeed.tech/tags/gemma.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [inference](<https://devfeed.tech/tags/inference.md>), [litert](<https://devfeed.tech/tags/litert.md>), [offline](<https://devfeed.tech/tags/offline.md>), [on-device](<https://devfeed.tech/tags/on-device.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [robotics](<https://devfeed.tech/tags/robotics.md>)

### AI overview

The article explains how to deploy Gemma models with LiteRT on a Raspberry Pi for local, real-time edge AI applications such as robotics. It highlights LiteRT-LM, CPU and GPU optimization, and reported performance figures for Gemma 4 E2B on Raspberry Pi 5.

### Source excerpt

Deploying secure, real-time Edge AI on Raspberry Pi is now simplified using LiteRT and lightweight Gemma open models. LiteRT optimizes CPU and GPU performance, delivering fast token speeds for models like Gemma4, enabling real-time local reasoning for robotics. Developers can quickly convert, quantize, and run these models using the lightweight LiteRT CLI tool. Support for Hailo AI accelerators is also coming very soon.

## Announcing ADK for Kotlin 1.0: Building Production-Ready AI Agents in Kotlin, Android, and Beyond

DevFeed: [Announcing ADK for Kotlin 1.0: Building Production-Ready AI Agents in Kotlin, Android, and Beyond](<https://devfeed.tech/articles/announcing-adk-for-kotlin-1-0-building-production-ready-ai-agents-in-kotlin-android-and-beyond-4204.md>)

Original publisher: [Read original article](<https://developers.googleblog.com/announcing-adk-for-kotlin-10-building-production-ready-ai-agents-in-kotlin-android-and-beyond/>)

Author: Guillaume Laforge

Published: 2026-09-12T11:04:33.891311Z

Content type: release

Language: en

Sources: [Google Developers Blog](<https://devfeed.tech/sources/google-developers-blog.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Kotlin](<https://devfeed.tech/topics/kotlin.md>), [AI Development](<https://devfeed.tech/topics/ai-development.md>), [Android](<https://devfeed.tech/topics/android.md>), [multiplatform](<https://devfeed.tech/topics/multiplatform.md>), [Orchestration](<https://devfeed.tech/topics/orchestration.md>), [Google](<https://devfeed.tech/topics/google.md>), [Persistence](<https://devfeed.tech/topics/persistence.md>), [Agent Skills](<https://devfeed.tech/topics/agent-skills.md>), [LiteRT](<https://devfeed.tech/topics/litert.md>), [Cloud](<https://devfeed.tech/topics/cloud.md>)

Tags: [agent-skill](<https://devfeed.tech/tags/agent-skill.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [android](<https://devfeed.tech/tags/android.md>), [building](<https://devfeed.tech/tags/building.md>), [database](<https://devfeed.tech/tags/database.md>), [development-kit](<https://devfeed.tech/tags/development-kit.md>), [firebase](<https://devfeed.tech/tags/firebase.md>), [incident](<https://devfeed.tech/tags/incident.md>), [kotlin](<https://devfeed.tech/tags/kotlin.md>), [multiplatform](<https://devfeed.tech/tags/multiplatform.md>), [orchestration](<https://devfeed.tech/tags/orchestration.md>), [persistence](<https://devfeed.tech/tags/persistence.md>), [production](<https://devfeed.tech/tags/production.md>)

### AI overview

Google announces the 1.0 general availability release of the Agent Development Kit (ADK) for Kotlin, a production-ready toolkit for building multi-agent applications with Kotlin, Java, and Android. Built on Kotlin Multiplatform, it provides feature parity with the ADK 1.0 Core and adds Android-first extensions for on-device agents with LiteRT-LM and ML Kit, hybrid cloud workflows through Firebase AI Logic, and state persistence with Room and AppSearch. The release also includes type-safe, compile-time function calling through KSP and declarative agent skills.

### Source excerpt

Google has officially released version 1.0 of the Agent Development Kit (ADK) for Kotlin, achieving full feature parity with the Python and Java ADK cores to enable idiomatic, multi-agent AI development. Built on Kotlin Multiplatform (KMP), the framework leverages Kotlin Symbol Processing (KSP) for zero-reflection, type-safe function calling, alongside advanced orchestration capabilities like human-in-the-loop workflows and context compaction. Additionally, the release introduces a robust suite of Android-first extensions, allowing mobile developers to integrate local models via LiteRT-LM, cloud reasoning through Firebase AI, session persistence using Room, and semantic memory powered by AppSearch.

## Codename One Adds On-Device AI and Loopback MCP Support

DevFeed: [Codename One Adds On-Device AI and Loopback MCP Support](<https://devfeed.tech/articles/on-device-ai-and-mcp-on-every-port-19421.md>)

Original publisher: [Read original article](<https://www.codenameone.com/blog/on-device-ai-mcp-loopback/>)

Author: Shai Almog

Published: 2026-08-02T00:00:00Z

Content type: release

Language: en

Sources: [CodeName One](<https://devfeed.tech/sources/codename-one.md>)

Topics: [On-device AI](<https://devfeed.tech/topics/on-device-ai.md>), [Model Context Protocol](<https://devfeed.tech/topics/model-context-protocol.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [LiteRT](<https://devfeed.tech/topics/litert.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [Mobile](<https://devfeed.tech/topics/mobile.md>), [supply-chain-security](<https://devfeed.tech/topics/supply-chain-security.md>)

Tags: [applications](<https://devfeed.tech/tags/applications.md>), [inference](<https://devfeed.tech/tags/inference.md>), [litert](<https://devfeed.tech/tags/litert.md>), [mcp](<https://devfeed.tech/tags/mcp.md>), [mobile](<https://devfeed.tech/tags/mobile.md>), [on-device](<https://devfeed.tech/tags/on-device.md>), [on-device-ai](<https://devfeed.tech/tags/on-device-ai.md>), [tensorflow-lite](<https://devfeed.tech/tags/tensorflow-lite.md>)

### AI overview

Codename One has added on-device vision, language, and LiteRT inference to its core, alongside a guarded loopback MCP transport for inspecting and operating applications. The article explains platform support, asynchronous OCR, model sessions, secure model downloads, and privacy limitations of local inference.

### Source excerpt

Codename One now exposes on-device vision, language, and LiteRT inference in the core, while a guarded loopback MCP transport lets an LLM inspect and drive real applications.

## Push V3: One Message From Your Server to Every Surface

DevFeed: [Push V3: One Message From Your Server to Every Surface](<https://devfeed.tech/articles/push-v3-one-message-from-your-server-to-every-surface-19457.md>)

Original publisher: [Read original article](<https://www.codenameone.com/blog/push-v3-new-cloud/>)

Author: Shai Almog

Published: 2026-07-31T00:00:00Z

Content type: release

Language: en

Sources: [CodeName One](<https://devfeed.tech/sources/codename-one.md>)

Topics: [releases](<https://devfeed.tech/topics/releases.md>), [Cloud](<https://devfeed.tech/topics/cloud.md>), [Mobile](<https://devfeed.tech/topics/mobile.md>), [API](<https://devfeed.tech/topics/api.md>), [LiteRT](<https://devfeed.tech/topics/litert.md>), [Model Context Protocol](<https://devfeed.tech/topics/model-context-protocol.md>)

Tags: [api](<https://devfeed.tech/tags/api.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [litert](<https://devfeed.tech/tags/litert.md>), [mcp-server](<https://devfeed.tech/tags/mcp-server.md>), [mobile](<https://devfeed.tech/tags/mobile.md>), [release](<https://devfeed.tech/tags/release.md>), [releases](<https://devfeed.tech/tags/releases.md>), [updates](<https://devfeed.tech/tags/updates.md>)

### AI overview

Codename One Push V3 adds typed messages, managed credentials, subscriptions, server-side segments, campaigns, analytics, and Surface updates. Existing push applications should test the new cloud endpoint before the planned cutover.

### Source excerpt

Codename One Push V3 adds typed messages, managed credentials, segmentation, analytics, and Surface updates. Existing push apps should test the new cloud endpoint before next week's cutover.

## Gesture Recognition Based on TFLite

DevFeed: [Gesture Recognition Based on TFLite](<https://devfeed.tech/articles/gesture-recognition-based-on-tflite-13765.md>)

Original publisher: [Read original article](<https://developer.espressif.com/blog/2026/04/gesture-recognition-based-on-tflite/>)

Author: John Lee

Published: 2026-04-10T00:00:00Z

Content type: tutorial

Language: en

Sources: [Blog on Developer Portal](<https://devfeed.tech/sources/blog-on-developer-portal.md>)

Topics: [LiteRT](<https://devfeed.tech/topics/litert.md>), [Espressif](<https://devfeed.tech/topics/espressif.md>), [model-deployment](<https://devfeed.tech/topics/model-deployment.md>), [Tensorflow](<https://devfeed.tech/topics/tensorflow.md>), [Keras](<https://devfeed.tech/topics/keras.md>), [On-device AI](<https://devfeed.tech/topics/on-device-ai.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [blog](<https://devfeed.tech/tags/blog.md>), [deep-learning](<https://devfeed.tech/tags/deep-learning.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [development-board](<https://devfeed.tech/tags/development-board.md>), [edge-ai](<https://devfeed.tech/tags/edge-ai.md>), [edge-computing](<https://devfeed.tech/tags/edge-computing.md>), [esp-idf](<https://devfeed.tech/tags/esp-idf.md>), [espressif](<https://devfeed.tech/tags/espressif.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [inference](<https://devfeed.tech/tags/inference.md>), [keras](<https://devfeed.tech/tags/keras.md>), [model-deployment](<https://devfeed.tech/tags/model-deployment.md>), [model-training](<https://devfeed.tech/tags/model-training.md>), [tensorflow-lite](<https://devfeed.tech/tags/tensorflow-lite.md>)

### AI overview

This tutorial demonstrates gesture recognition on Espressif SoCs using TensorFlow Lite Micro. It covers data collection, model training, conversion for TFLite Micro, and deployment with C++ code for model loading, preprocessing, and inference.

### Source excerpt

This article demonstrates how to implement gesture recognition using TensorFlow Lite Micro on Espressif SoCs. It covers the complete workflow from data collection and model training to model deployment, showcasing TensorFlow Lite Micro's applications in edge AI.

## Gradium Phonon: On-Device TTS for Consumer Apps, NPCs, and Offline Products

DevFeed: [Gradium Phonon: On-Device TTS for Consumer Apps, NPCs, and Offline Products](<https://devfeed.tech/articles/gradium-phonon-on-device-tts-for-consumer-apps-npcs-and-offline-products-81226.md>)

Original publisher: [Read original article](<https://gradium.ai/blog/gradium-phonon>)

Author: Gradium

Published: 2026-04-03T00:00:00Z

Content type: release

Language: en

Sources: [Gradium](<https://devfeed.tech/sources/gradium.md>)

Topics: [Local-First](<https://devfeed.tech/topics/local-first.md>), [LiteRT](<https://devfeed.tech/topics/litert.md>), [iOS](<https://devfeed.tech/topics/ios.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [announcement](<https://devfeed.tech/tags/announcement.md>), [app](<https://devfeed.tech/tags/app.md>), [consumer-app-voice-ai](<https://devfeed.tech/tags/consumer-app-voice-ai.md>), [edge-tts-launch](<https://devfeed.tech/tags/edge-tts-launch.md>), [gradium-phonon](<https://devfeed.tech/tags/gradium-phonon.md>), [npc-voice-ai](<https://devfeed.tech/tags/npc-voice-ai.md>), [offline-text-to-speech](<https://devfeed.tech/tags/offline-text-to-speech.md>), [on-device](<https://devfeed.tech/tags/on-device.md>), [on-device-tts](<https://devfeed.tech/tags/on-device-tts.md>), [speech](<https://devfeed.tech/tags/speech.md>), [text](<https://devfeed.tech/tags/text.md>), [text-to-speech](<https://devfeed.tech/tags/text-to-speech.md>), [tts](<https://devfeed.tech/tags/tts.md>)

### AI overview

Gradium Phonon is an on-device text-to-speech model for Android and iOS that runs on CPU without a network connection. The article describes its fixed per-device licensing, offline and privacy benefits, and trade-offs against Gradium's cloud API, including its narrower scope of one voice, target language, and content type per model. Phonon is available through a private beta.

### Source excerpt

Announcing Gradium Phonon, our new on-device text-to-speech model designed for consumer apps, NPCs, and offline products.

## What's new in TensorFlow 2.20

DevFeed: [What's new in TensorFlow 2.20](<https://devfeed.tech/articles/what-s-new-in-tensorflow-2-20-7424.md>)

Original publisher: [Read original article](<https://blog.tensorflow.org/2025/08/whats-new-in-tensorflow-2-20.html>)

Author: TensorFlow Blog (noreply@blogger.com)

Published: 2025-08-19T16:00:00Z

Content type: news

Language: en

Sources: [The TensorFlow Blog](<https://devfeed.tech/sources/the-tensorflow-blog.md>)

Topics: [LiteRT](<https://devfeed.tech/topics/litert.md>), [On-device AI](<https://devfeed.tech/topics/on-device-ai.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [Latency](<https://devfeed.tech/topics/latency.md>), [IO](<https://devfeed.tech/topics/io.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [data](<https://devfeed.tech/topics/data.md>), [Google](<https://devfeed.tech/topics/google.md>), [Cloud](<https://devfeed.tech/topics/cloud.md>), [Python](<https://devfeed.tech/topics/python.md>), [C++](<https://devfeed.tech/topics/c-plus-plus.md>), [Kotlin](<https://devfeed.tech/topics/kotlin.md>)

Tags: [announcement](<https://devfeed.tech/tags/announcement.md>), [c-plus-plus](<https://devfeed.tech/tags/c-plus-plus.md>), [data](<https://devfeed.tech/tags/data.md>), [google](<https://devfeed.tech/tags/google.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [inference](<https://devfeed.tech/tags/inference.md>), [io](<https://devfeed.tech/tags/io.md>), [kotlin](<https://devfeed.tech/tags/kotlin.md>), [latency](<https://devfeed.tech/tags/latency.md>), [litert](<https://devfeed.tech/tags/litert.md>), [npu](<https://devfeed.tech/tags/npu.md>), [on-device](<https://devfeed.tech/tags/on-device.md>), [python](<https://devfeed.tech/tags/python.md>), [tensorflow](<https://devfeed.tech/tags/tensorflow.md>), [tensorflow-core](<https://devfeed.tech/tags/tensorflow-core.md>), [tensorflow-lite](<https://devfeed.tech/tags/tensorflow-lite.md>)

### AI overview

TensorFlow 2.20 introduces the independent LiteRT repository as the future home for on-device inference, replacing the deprecated tf.lite module. LiteRT adds Kotlin and C++ APIs, improves NPU and GPU acceleration, and supports lower-latency inference with fewer memory copies. The release also adds tf.data input-pipeline warm-up controls and makes Google Cloud Storage support optional.

### Source excerpt

Posted by the TensorFlow team TensorFlow 2.20 has been released! For ongoing updates related to the multi-backend Keras, please note that all news and releases, starting with Keras 3.0, are now published directly on keras.io. You can find a complete list of all changes in the full release notes on GitHub. tf.lite is being replaced by LiteRT The tf.lite module will be deprecated with development for on-device inference moving to a new, independent repository: LiteRT. The new APIs are available in Kotlin and C++. This code base will decouple from the TensorFlow repository and tf.lite will be removed from future TensorFlow Python packages, so we encourage migration of projects to LiteRT to receive the latest updates. More details to follow. As announced at Google I/O '25, LiteRT improves upon TFLite, particularly for NPU and GPU hardware acceleration and performance for on-device ML and AI applications. LiteRT provides a unified interface for Neural Processing Units (NPUs), removing the need to navigate vendor-specific compilers or libraries. This approach avoids many device-specific complications, boosts performance for real-time and large-model inference, and minimizes memory copies through zero-copy hardware buffer usage. For more information on the new repository and to sign up for the NPU Early Access Program, please reach out to the team at g.co/ai/LiteRT-NPU-EAP. Faster input pipeline warm-up with tf.data To help reduce latency, especially the time it takes for your model to process the first element of a dataset, we've added autotune.min_parallelism in tf.data.Options. This new option allows asynchronous dataset operations like .map and .batch to immediately start with a specified minimum level of parallelism, speeding up the initial warm-up time for your input pipelines. Changes to I/O GCS filesystem package The tensorflow-io-gcs-filesystem package for Google Cloud Storage support is now optional. Previously, it was installed, by default, with TensorFlow. If y

## What's new in TensorFlow 2.19

DevFeed: [What's new in TensorFlow 2.19](<https://devfeed.tech/articles/what-s-new-in-tensorflow-2-19-7421.md>)

Original publisher: [Read original article](<https://blog.tensorflow.org/2025/03/whats-new-in-tensorflow-2-19.html>)

Author: TensorFlow Blog (noreply@blogger.com)

Published: 2025-03-13T16:00:00Z

Content type: release

Language: en

Sources: [The TensorFlow Blog](<https://devfeed.tech/sources/the-tensorflow-blog.md>)

Topics: [Tensorflow](<https://devfeed.tech/topics/tensorflow.md>), [LiteRT](<https://devfeed.tech/topics/litert.md>), [releases](<https://devfeed.tech/topics/releases.md>), [API](<https://devfeed.tech/topics/api.md>), [C++](<https://devfeed.tech/topics/c-plus-plus.md>), [Release notes](<https://devfeed.tech/topics/release-notes.md>), [Keras](<https://devfeed.tech/topics/keras.md>), [migration](<https://devfeed.tech/topics/migration.md>)

Tags: [2025](<https://devfeed.tech/tags/2025.md>), [announcement](<https://devfeed.tech/tags/announcement.md>), [api](<https://devfeed.tech/tags/api.md>), [c-plus-plus](<https://devfeed.tech/tags/c-plus-plus.md>), [keras](<https://devfeed.tech/tags/keras.md>), [libentensorflow](<https://devfeed.tech/tags/libentensorflow.md>), [litert](<https://devfeed.tech/tags/litert.md>), [migration-guide](<https://devfeed.tech/tags/migration-guide.md>), [pypi](<https://devfeed.tech/tags/pypi.md>), [release](<https://devfeed.tech/tags/release.md>), [release-notes](<https://devfeed.tech/tags/release-notes.md>), [tensorflow](<https://devfeed.tech/tags/tensorflow.md>), [tensorflow-core](<https://devfeed.tech/tags/tensorflow-core.md>), [tensorflowlite](<https://devfeed.tech/tags/tensorflowlite.md>)

### AI overview

TensorFlow 2.19 introduces C++ API changes in LiteRT, adds bfloat16 support for the tflite casting operation, and deprecates tf.lite.Interpreter in favor of ai_edge_litert.interpreter. The release also stops publishing libtensorflow packages, although they remain extractable from the PyPI package. Updates for multi-backend Keras are directed to keras.io.

### Source excerpt

Posted by the TensorFlow team TensorFlow 2.19 has been released! Highlights of this release include changes to the C++ API in LiteRT, bfloat16 support for tflite casting, discontinue of releasing libtensorflow packages. Learn more by reading the full release notes. Note: Release updates on the new multi-backend Keras will be published on keras.io, starting with Keras 3.0. For more information, please see https://keras.io/keras_3/. TensorFlow Core LiteRT The public constants tflite::Interpreter:kTensorsReservedCapacity and tflite::Interpreter:kTensorsCapacityHeadroom are now const references, rather than constexpr compile-time constants. (This is to enable better API compatibility for TFLite in Play services while preserving the implementation flexibility to change the values of these constants in the future.) TF-Lite tfl.Cast op is now supporting bfloat16 in the runtime kernel. tf.lite.Interpreter gives a deprecation warning redirecting to its new location at ai_edge_litert.interpreter, as the API tf.lite.Interpreter will be deleted in TF 2.20. See the migration guide for details. Libtensorflow We have stopped publishing libtensorflow packages but it can still be unpacked from the PyPI package.

## What's new in TensorFlow 2.18

DevFeed: [What's new in TensorFlow 2.18](<https://devfeed.tech/articles/what-s-new-in-tensorflow-2-18-7415.md>)

Original publisher: [Read original article](<https://blog.tensorflow.org/2024/10/whats-new-in-tensorflow-218.html>)

Author: TensorFlow Blog (noreply@blogger.com)

Published: 2024-10-28T19:00:00Z

Content type: release

Language: en

Sources: [The TensorFlow Blog](<https://devfeed.tech/sources/the-tensorflow-blog.md>)

Topics: [Tensorflow](<https://devfeed.tech/topics/tensorflow.md>), [releases](<https://devfeed.tech/topics/releases.md>), [LiteRT](<https://devfeed.tech/topics/litert.md>), [NumPy](<https://devfeed.tech/topics/numpy.md>), [CUDA](<https://devfeed.tech/topics/cuda.md>), [cudnn](<https://devfeed.tech/topics/cudnn.md>), [NCCL](<https://devfeed.tech/topics/nccl.md>), [Google](<https://devfeed.tech/topics/google.md>)

Tags: [announcement](<https://devfeed.tech/tags/announcement.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [cudnn](<https://devfeed.tech/tags/cudnn.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [litert](<https://devfeed.tech/tags/litert.md>), [migration](<https://devfeed.tech/tags/migration.md>), [nccl](<https://devfeed.tech/tags/nccl.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [nvidia-rtx](<https://devfeed.tech/tags/nvidia-rtx.md>), [python](<https://devfeed.tech/tags/python.md>), [release](<https://devfeed.tech/tags/release.md>), [releases](<https://devfeed.tech/tags/releases.md>), [reproducible-builds](<https://devfeed.tech/tags/reproducible-builds.md>), [tensorflow](<https://devfeed.tech/tags/tensorflow.md>), [tensorflow-core](<https://devfeed.tech/tags/tensorflow-core.md>)

### AI overview

TensorFlow 2.18 introduces NumPy 2.0 compatibility updates, transitions TFLite development to the LiteRT repository, and adds hermetic CUDA, cuDNN, and NCCL dependencies for more reproducible source builds. Binary packages add dedicated kernels for compute capability 8.9 GPUs, including NVIDIA RTX 40 series, L4, and L40, while dropping precompiled support for Maxwell GPUs.

### Source excerpt

Posted by the TensorFlow team TensorFlow 2.18 has been released! Highlights of this release (and 2.17) include NumPy 2.0, LiteRT repository, CUDA Update, Hermetic CUDA and more. For the full release notes, please click here. Note: Release updates on the new multi-backend Keras will be published on keras.io, starting with Keras 3.0. For more information, please see https://keras.io/keras_3/. TensorFlow Core NumPy 2.0 The upcoming TensorFlow 2.18 release will include support for NumPy 2.0. While the majority of TensorFlow APIs will function seamlessly with NumPy 2.0, this may break some edge cases of usage, e.g., out-of-boundary conversion errors and numpy scalar representation errors. You can consult the following common solutions. Note that NumPy's type promotion rules have been changed (See NEP 50 for details). This may change the precision at which computations happen, leading either to type errors or to numerical changes to results. Please see the NumPy 2 migration guide. We've updated some TensorFlow tensor APIs to maintain compatibility with NumPy 2.0 while preserving the out-of-boundary conversion behavior in NumPy 1.x. LiteRT Repository We're making some changes to how LiteRT (formerly known as TFLite) is developed. Over the coming months, we'll be gradually transitioning TFLite's codebase to LiteRT. Once the migration is complete, we'll start accepting contributions directly through the LiteRT repository. There will no longer be any binary TFLite releases and developers should switch to LiteRT for the latest updates. Hermetic CUDA If you build TensorFlow from source, Bazel will now download specific versions of CUDA, CUDNN and NCCL distributions, and then use those tools as dependencies in various Bazel targets. This enables more reproducible builds for Google ML projects and supported CUDA versions because the build no longer relies on the locally installed versions. More details are provided here. CUDA Update TensorFlow binary distributions now ship with d

## Faster Dynamically Quantized Inference with XNNPack

DevFeed: [Faster Dynamically Quantized Inference with XNNPack](<https://devfeed.tech/articles/faster-dynamically-quantized-inference-with-xnnpack-7410.md>)

Original publisher: [Read original article](<https://blog.tensorflow.org/2024/04/faster-dynamically-quantized-inference-with-xnnpack.html>)

Author: TensorFlow Blog (noreply@blogger.com)

Published: 2024-04-09T16:00:00Z

Content type: article

Language: en

Sources: [The TensorFlow Blog](<https://devfeed.tech/sources/the-tensorflow-blog.md>)

Topics: [Inference](<https://devfeed.tech/topics/inference.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [LiteRT](<https://devfeed.tech/topics/litert.md>), [cpu](<https://devfeed.tech/topics/cpu.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [model-deployment](<https://devfeed.tech/topics/model-deployment.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [announcement](<https://devfeed.tech/tags/announcement.md>), [article](<https://devfeed.tech/tags/article.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [inference](<https://devfeed.tech/tags/inference.md>), [inference-performance](<https://devfeed.tech/tags/inference-performance.md>), [learn](<https://devfeed.tech/tags/learn.md>), [ml](<https://devfeed.tech/tags/ml.md>), [performance](<https://devfeed.tech/tags/performance.md>), [precision](<https://devfeed.tech/tags/precision.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [range](<https://devfeed.tech/tags/range.md>), [scale](<https://devfeed.tech/tags/scale.md>), [tensorflow](<https://devfeed.tech/tags/tensorflow.md>), [tensorflow-lite](<https://devfeed.tech/tags/tensorflow-lite.md>), [tensors](<https://devfeed.tech/tags/tensors.md>)

### AI overview

This article explains how dynamic range quantization for XNNPack's Fully Connected and Convolution 2D operators improves CPU inference performance in TensorFlow Lite. The approach quantizes weights to 8-bit integers during model conversion and dynamically quantizes activations during inference, delivering most of the performance benefits of full quantization while retaining higher overall accuracy. The article reports a fourfold performance improvement over the single-precision baseline and describes how this can enable AI-powered features on older and lower-tier devices.

### Source excerpt

Posted by Alan Kelly, Software Engineer We are excited to announce that XNNPack's Fully Connected and Convolution 2D operators now support dynamic range quantization. XNNPack is TensorFlow Lite's CPU backend and CPUs deliver the widest reach for ML inference and remain the default target for TensorFlow Lite. Consequently, improving CPU inference performance is a top priority. We quadrupled inference performance in TensorFlow Lite's XNNPack backend compared to the single precision baseline by adding support for dynamic range quantization to the Fully Connected and Convolution operators. This means that more AI powered features may be deployed to older and lower tier devices. Previously, XNNPack offered users the choice between either full integer quantization, where the weights and activations are stored as signed 8-bit integers, or half-precision (fp16) or single-precision (fp32) floating-point inference. In this article we demonstrate the benefits of dynamic range quantization. Dynamic Range Quantization Dynamically quantized models are similar to fully-quantized models in that the weights for the Fully Connected and Convolution operators are quantized to 8-bit integers during model conversion. All other tensors are not quantized, they remain as float32 tensors. During model inference, the floating-point layer activations are converted to 8-bit integers before being passed to the Fully Connected and Convolution operators. The quantization parameters (the zero point and scale) for each row of the activation tensor are calculated dynamically based on the observed range of activations. This maximizes the accuracy of the quantization process as the activations make full use of the 8 quantized bits. In fully-quantized models, these parameters are fixed during model conversion, based on the range of the activation values observed using a representative dataset. The second difference between full quantization and dynamic range quantization is that the output of the Fully

## Half-precision Inference Doubles On-Device Inference Performance

DevFeed: [Half-precision Inference Doubles On-Device Inference Performance](<https://devfeed.tech/articles/half-precision-inference-doubles-on-device-inference-performance-7396.md>)

Original publisher: [Read original article](<https://blog.tensorflow.org/2023/11/half-precision-inference-doubles-on-device-inference-performance.html>)

Author: TensorFlow Blog (noreply@blogger.com)

Published: 2023-11-29T18:00:00Z

Content type: release

Language: en

Sources: [The TensorFlow Blog](<https://devfeed.tech/sources/the-tensorflow-blog.md>)

Topics: [Inference](<https://devfeed.tech/topics/inference.md>), [On-device AI](<https://devfeed.tech/topics/on-device-ai.md>), [LiteRT](<https://devfeed.tech/topics/litert.md>), [cpu](<https://devfeed.tech/topics/cpu.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [announcement](<https://devfeed.tech/tags/announcement.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [inference](<https://devfeed.tech/tags/inference.md>), [inference-performance](<https://devfeed.tech/tags/inference-performance.md>), [learn](<https://devfeed.tech/tags/learn.md>), [on-device](<https://devfeed.tech/tags/on-device.md>), [tensorflow-lite](<https://devfeed.tech/tags/tensorflow-lite.md>)

### AI overview

TensorFlow Lite and XNNPack add generally available FP16 inference for supported ARM CPUs, reporting close to 2x speedups for floating-point models in production and benchmarked mobile and laptop environments.

### Source excerpt

Posted by Marat Dukhan and Frank Barchard, Software Engineers CPUs deliver the widest reach for ML inference and remain the default target for TensorFlow Lite. Consequently, improving CPU inference performance is a top priority, and we are excited to announce that we doubled floating-point inference performance in TensorFlow Lite's XNNPack backend by enabling half-precision inference on ARM CPUs. This means that more AI powered features may be deployed to older and lower tier devices. Traditionally, TensorFlow Lite supported two kinds of numerical computations in machine learning models: a) floating-point using IEEE 754 single-precision (32-bit) format and b) quantized using low-precision integers. While single-precision floating-point numbers provide maximum flexibility and ease of use, they come at the cost of 4X overhead in storage and memory and exhibit a performance overhead compared to 8-bit integer computations. In contrast, half-precision (FP16) floating-point numbers pose an interesting alternative balancing ease-of-use and performance: the processor needs to transfer twice fewer bytes and each vector operation produces twice more elements. By virtue of this property, FP16 inference paves the way for 2X speedup for floating-point models compared to the traditional FP32 way. For a long time FP16 inference on CPUs primarily remained a research topic, as the lack of hardware support for FP16 computations limited production use-cases. However, around 2017 new mobile chipsets started to include support for native FP16 computations, and by now most mobile phones, both on the high-end and the low-end. Building upon this broad availability, we are pleased to announce the general availability for half-precision inference in TensorFlow Lite and XNNPack. Performance Improvements Half-precision inference has already been battle-tested in production across Google Assistant, Google Meet, YouTube, and ML Kit, and demonstrated close to 2X speedups across a wide range of ne

[Next page](<https://devfeed.tech/topics/litert.md?cursor=WyIyMDIzLTExLTI5VDE4OjAwOjAwKzAwOjAwIiwgIjllOWY5ZGI2LWJiNDktNGU0NC04OWE1LTEyOTIxMzEwZGNjNSJd>)