# gaudi

Published articles for gaudi.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## 🚀 Accelerating LLM Inference with TGI on Intel Gaudi

DevFeed: [🚀 Accelerating LLM Inference with TGI on Intel Gaudi](<https://devfeed.tech/articles/accelerating-llm-inference-with-tgi-on-intel-gaudi-7289.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/intel-gaudi-backend-for-tgi>)

Author: Baptiste Colle; Régis Pierrard; Ilyas Moutawwakil; Ella Charlaix; Ke Ding

Published: 2025-03-28T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [tgi](<https://devfeed.tech/topics/tgi.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [intel](<https://devfeed.tech/topics/intel.md>), [model-deployment](<https://devfeed.tech/topics/model-deployment.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>)

Tags: [aws](<https://devfeed.tech/tags/aws.md>), [batching](<https://devfeed.tech/tags/batching.md>), [dell](<https://devfeed.tech/tags/dell.md>), [docker](<https://devfeed.tech/tags/docker.md>), [gaudi](<https://devfeed.tech/tags/gaudi.md>), [gemma](<https://devfeed.tech/tags/gemma.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [ibm](<https://devfeed.tech/tags/ibm.md>), [inference](<https://devfeed.tech/tags/inference.md>), [intel](<https://devfeed.tech/tags/intel.md>), [llm](<https://devfeed.tech/tags/llm.md>), [mixtral](<https://devfeed.tech/tags/mixtral.md>), [partnerships](<https://devfeed.tech/tags/partnerships.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [tgi](<https://devfeed.tech/tags/tgi.md>)

### AI overview

This article announces the integration of Intel Gaudi support into the main TGI codebase through TGI's multi-backend architecture. It describes deployment options across Gaudi hardware generations, production features such as dynamic batching and streamed responses, support for multiple LLMs, multi-card inference, vision-language models, FP8 precision, and an official Docker image.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.

## Faster assisted generation support for Intel Gaudi

DevFeed: [Faster assisted generation support for Intel Gaudi](<https://devfeed.tech/articles/faster-assisted-generation-support-for-intel-gaudi-7108.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/assisted-generation-support-gaudi>)

Author: Haim Barad; Neha Raste; Tien Pei Chou

Published: 2024-06-04T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [intel](<https://devfeed.tech/topics/intel.md>), [text-generation](<https://devfeed.tech/topics/text-generation.md>), [Optimization](<https://devfeed.tech/topics/optimization.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [Cache](<https://devfeed.tech/topics/cache.md>), [Transformer](<https://devfeed.tech/topics/transformer.md>), [optimum](<https://devfeed.tech/topics/optimum.md>), [hugging face](<https://devfeed.tech/topics/hugging-face.md>)

Tags: [accelerate](<https://devfeed.tech/tags/accelerate.md>), [ai](<https://devfeed.tech/tags/ai.md>), [diffusers](<https://devfeed.tech/tags/diffusers.md>), [gaudi](<https://devfeed.tech/tags/gaudi.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [inference](<https://devfeed.tech/tags/inference.md>), [intel](<https://devfeed.tech/tags/intel.md>), [latency](<https://devfeed.tech/tags/latency.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [optimum](<https://devfeed.tech/tags/optimum.md>), [partnerships](<https://devfeed.tech/tags/partnerships.md>), [techniques](<https://devfeed.tech/tags/techniques.md>), [text-generation](<https://devfeed.tech/tags/text-generation.md>), [transformers](<https://devfeed.tech/tags/transformers.md>)

### AI overview

The article explains how assisted decoding and speculative sampling were adapted and optimized for Intel Gaudi processors. Integrated into Optimum Habana, these techniques use draft and target models, KV caching, and quantized models to accelerate text generation while preserving the target model's sampling quality.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.

## Building Cost-Efficient Enterprise RAG applications with Intel Gaudi 2 and Intel Xeon

DevFeed: [Building Cost-Efficient Enterprise RAG applications with Intel Gaudi 2 and Intel Xeon](<https://devfeed.tech/articles/building-cost-efficient-enterprise-rag-applications-with-intel-gaudi-2-and-intel-xeon-7151.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/cost-efficient-rag-applications-with-intel>)

Author: Julien Simon; Haihao Shen; Antony Vance Jeyaraj; Matrix Yao; Leon Lv; Greg Serochi; Deb Bharadwaj; Ke Ding

Published: 2024-05-09T00:00:00Z

Content type: tutorial

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [AI search](<https://devfeed.tech/topics/ai-search.md>), [Language models](<https://devfeed.tech/topics/language-models.md>)

Tags: [cost](<https://devfeed.tech/tags/cost.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [gaudi](<https://devfeed.tech/tags/gaudi.md>), [inference](<https://devfeed.tech/tags/inference.md>), [intel](<https://devfeed.tech/tags/intel.md>), [langchain](<https://devfeed.tech/tags/langchain.md>), [llm](<https://devfeed.tech/tags/llm.md>), [partnerships](<https://devfeed.tech/tags/partnerships.md>), [rag](<https://devfeed.tech/tags/rag.md>), [redis](<https://devfeed.tech/tags/redis.md>)

### AI overview

A tutorial on building a cost-efficient enterprise RAG application with LangChain, Redis, Intel Granite Rapids CPUs, and Intel Gaudi 2 accelerators.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.