# 🚀 Accelerating LLM Inference with TGI on Intel Gaudi

DevFeed: [🚀 Accelerating LLM Inference with TGI on Intel Gaudi](<https://devfeed.tech/articles/accelerating-llm-inference-with-tgi-on-intel-gaudi-7289.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/intel-gaudi-backend-for-tgi>)

Author: Baptiste Colle; Régis Pierrard; Ilyas Moutawwakil; Ella Charlaix; Ke Ding

Published: 2025-03-28T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [tgi](<https://devfeed.tech/topics/tgi.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [intel](<https://devfeed.tech/topics/intel.md>), [model-deployment](<https://devfeed.tech/topics/model-deployment.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>)

Tags: [aws](<https://devfeed.tech/tags/aws.md>), [batching](<https://devfeed.tech/tags/batching.md>), [dell](<https://devfeed.tech/tags/dell.md>), [docker](<https://devfeed.tech/tags/docker.md>), [gaudi](<https://devfeed.tech/tags/gaudi.md>), [gemma](<https://devfeed.tech/tags/gemma.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [ibm](<https://devfeed.tech/tags/ibm.md>), [inference](<https://devfeed.tech/tags/inference.md>), [intel](<https://devfeed.tech/tags/intel.md>), [llm](<https://devfeed.tech/tags/llm.md>), [mixtral](<https://devfeed.tech/tags/mixtral.md>), [partnerships](<https://devfeed.tech/tags/partnerships.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [tgi](<https://devfeed.tech/tags/tgi.md>)

## AI overview

This article announces the integration of Intel Gaudi support into the main TGI codebase through TGI's multi-backend architecture. It describes deployment options across Gaudi hardware generations, production features such as dynamic batching and streamed responses, support for multiple LLMs, multi-card inference, vision-language models, FP8 precision, and an official Docker image.

## Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.