# On-Device AI Series (Part 5): LiteRT-LM

DevFeed: [On-Device AI Series (Part 5): LiteRT-LM](<https://devfeed.tech/articles/on-device-ai-series-part-5-litert-lm-22949.md>)

Original publisher: [Read original article](<https://proandroiddev.com/on-device-ai-series-part-5-litert-lm-d6c23b102094?source=rss----c72404660798---4>)

Author: Oğuzhan Aslan

Published: 2026-09-14T05:59:12Z

Content type: tutorial

Language: en

Sources: [ProAndroidDev - Medium](<https://devfeed.tech/sources/proandroiddev-medium.md>)

Topics: [LiteRT](<https://devfeed.tech/topics/litert.md>), [On-device AI](<https://devfeed.tech/topics/on-device-ai.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [PyTorch](<https://devfeed.tech/topics/pytorch.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [android](<https://devfeed.tech/tags/android.md>), [android-development](<https://devfeed.tech/tags/android-development.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [inference](<https://devfeed.tech/tags/inference.md>), [litert](<https://devfeed.tech/tags/litert.md>), [llama](<https://devfeed.tech/tags/llama.md>), [llm](<https://devfeed.tech/tags/llm.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [on-device](<https://devfeed.tech/tags/on-device.md>), [on-device-ai](<https://devfeed.tech/tags/on-device-ai.md>), [programming](<https://devfeed.tech/tags/programming.md>), [pytorch](<https://devfeed.tech/tags/pytorch.md>), [qwen](<https://devfeed.tech/tags/qwen.md>), [software-development](<https://devfeed.tech/tags/software-development.md>), [streaming](<https://devfeed.tech/tags/streaming.md>)

## AI overview

This tutorial explains LiteRT-LM for running large language models on-device. It covers the Engine/Session API, streaming output, system prompts, tool calling, multimodal inputs, thinking mode, and CPU-versus-GPU benchmarking. The article also discusses tradeoffs involving privacy, network independence, latency, memory, sampling configuration, and model capability compared with cloud APIs.

## Source excerpt

Put your phone in airplane mode. Open the app, type a question, and watch the answer arrive one token at a time -- no spinner waiting on a network round-trip, no API key, no per-token bill, and nothing you typed ever leaving the device. LiteRT-LM removes the genuinely hard parts of running an LLM on-device -- KV-cache management, token streaming, backend selection -- but it doesn't remove your job so much as relocate it. What's left on your plate is a short, specific list: sizing a combined input+output token budget, owning your own sampling defaults, hand-building system prompts and tool calling out of raw text, and one native-library collision that presents as a SIGSEGV rather than a build error. Know those going in and the API itself is a clean three-step pattern. We'll get there in that order: Why you'd choose this runtime and what it costs you versus the cloud. The Engine/Session model you need to read the code at all. Real implementation samples -- streaming, system prompts and tool calling, multimodal inputs, thinking mode, and CPU-vs-GPU benchmarking. The anti-patterns to avoid. A developer-friendliness rating on the same rubric as Parts 1-4. Why Use LiteRT-LM? You reach for LiteRT-LM instead of hand-rolling generation on top of raw LiteRT when: You need multi-turn conversation, not single-shot inference -- session state and KV-cache bookkeeping are handled for you, and resetting a conversation is a session swap, not a model reload. You need streaming output -- token-by-token delivery for a responsive chat UI, instead of a blocking call that returns everything at once. You're choosing between CPU and GPU per device -- the explicit backend parameter turns that into a runtime decision instead of a build-time guess. You want a pre-converted model without doing your own PyTorch-to-LiteRT conversion work -- the Model Zoo covers Gemma, Qwen, Llama, and more out of the box. You're willing to own sampling -- the engine won't pick sane decoding defaults for you; that's on the