# tgi

A Hugging Face toolkit for deploying and serving large language models with high-performance text generation.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Introducing the Palmyra-mini family: Powerful, lightweight, and ready to reason!

DevFeed: [Introducing the Palmyra-mini family: Powerful, lightweight, and ready to reason!](<https://devfeed.tech/articles/introducing-the-palmyra-mini-family-powerful-lightweight-and-ready-to-reason-7060.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/Writer/announcing-palmyra-mini>)

Author: Rakshith; Tom Peres

Published: 2025-09-11T20:04:44Z

Content type: news

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [Inference](<https://devfeed.tech/topics/inference.md>), [Chain-of-thought](<https://devfeed.tech/topics/chain-of-thought.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [qwen](<https://devfeed.tech/topics/qwen.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [sglang](<https://devfeed.tech/topics/sglang.md>), [tgi](<https://devfeed.tech/topics/tgi.md>), [vllm](<https://devfeed.tech/topics/vllm.md>)

Tags: [announce](<https://devfeed.tech/tags/announce.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [chain-of-thought](<https://devfeed.tech/tags/chain-of-thought.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [inference](<https://devfeed.tech/tags/inference.md>), [models](<https://devfeed.tech/tags/models.md>), [open](<https://devfeed.tech/tags/open.md>), [qwen](<https://devfeed.tech/tags/qwen.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [release](<https://devfeed.tech/tags/release.md>), [rl](<https://devfeed.tech/tags/rl.md>), [sglang](<https://devfeed.tech/tags/sglang.md>), [tgi](<https://devfeed.tech/tags/tgi.md>), [vllm](<https://devfeed.tech/tags/vllm.md>)

### AI overview

WRITER announces three open Palmyra-mini models ranging from 1.5B to 1.7B parameters: a lightweight base model and two reasoning variants. The models target efficient inference across varied applications, with GGUF and MLX quantizations available. The article reports benchmark results, describes Chain of Thought training for the reasoning variants, and discusses inference-framework compatibility and reinforcement-learning trade-offs.

### Source excerpt

The team at WRITER is thrilled to announce the release of three new open models in the Palmyra-mini family. These models are designed to be powerful, lightweight, and highly performant for their size (1.5B to 1.7B), making them ideal for a wide range of applications with efficient inference. - palmyra-mini: A powerful, lightweight non-thinking base model. - palmyra-mini-thinking-a: A specialized variant optimized for complex reasoning and logic.

## The Transformers Library: standardizing model definitions

DevFeed: [The Transformers Library: standardizing model definitions](<https://devfeed.tech/articles/the-transformers-library-standardizing-model-definitions-7535.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/transformers-model-definition>)

Author: Lysandre; Arthur Zucker; Pedro Cuenca; Julien Chaumond

Published: 2025-05-15T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [Transformer](<https://devfeed.tech/topics/transformer.md>), [LLMs](<https://devfeed.tech/topics/llms.md>), [interoperability](<https://devfeed.tech/topics/interoperability.md>), [sglang](<https://devfeed.tech/topics/sglang.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [llama.cpp](<https://devfeed.tech/topics/llama-cpp.md>), [tgi](<https://devfeed.tech/topics/tgi.md>), [MLX](<https://devfeed.tech/topics/mlx.md>)

Tags: [announcement](<https://devfeed.tech/tags/announcement.md>), [community](<https://devfeed.tech/tags/community.md>), [interoperability](<https://devfeed.tech/tags/interoperability.md>), [library](<https://devfeed.tech/tags/library.md>), [llama-cpp](<https://devfeed.tech/tags/llama-cpp.md>), [llms](<https://devfeed.tech/tags/llms.md>), [mlx](<https://devfeed.tech/tags/mlx.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [sglang](<https://devfeed.tech/tags/sglang.md>), [tgi](<https://devfeed.tech/tags/tgi.md>), [transformers](<https://devfeed.tech/tags/transformers.md>), [vllm](<https://devfeed.tech/tags/vllm.md>), [vlms](<https://devfeed.tech/tags/vlms.md>)

### AI overview

The article presents Transformers as a standard library for defining and supporting machine learning model architectures. It describes its broad ecosystem integrations, including training frameworks and inference engines, and highlights interoperability with vLLM, SGLang, TGI, llama.cpp, and MLX.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.

## Welcome Llama 4 Maverick & Scout on Hugging Face

DevFeed: [Welcome Llama 4 Maverick & Scout on Hugging Face](<https://devfeed.tech/articles/welcome-llama-4-maverick-scout-on-hugging-face-7338.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/llama4-release>)

Author: ben burtenshaw; Vaibhav Srivastav; Pedro Cuenca; Clem 🤗; Rajat Arya; Jared Sulzdorf; Lysandre

Published: 2025-04-05T00:00:00Z

Content type: release

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [llama](<https://devfeed.tech/topics/llama.md>), [hugging face](<https://devfeed.tech/topics/hugging-face.md>), [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [multimodal](<https://devfeed.tech/topics/multimodal.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>), [tgi](<https://devfeed.tech/topics/tgi.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [model-deployment](<https://devfeed.tech/topics/model-deployment.md>)

Tags: [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [hugging-face](<https://devfeed.tech/tags/hugging-face.md>), [inference](<https://devfeed.tech/tags/inference.md>), [llama](<https://devfeed.tech/tags/llama.md>), [meta](<https://devfeed.tech/tags/meta.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [tgi](<https://devfeed.tech/tags/tgi.md>), [transformers](<https://devfeed.tech/tags/transformers.md>)

### AI overview

Hugging Face announces the release of Meta's Llama 4 Maverick and Scout, natively multimodal models built with an autoregressive Mixture-of-Experts architecture. The article describes their parameter sizes, expert counts, training data, language support, quantization options, licensing, and integration with the Hugging Face Hub, Transformers, TRL, and TGI.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.

## 🚀 Accelerating LLM Inference with TGI on Intel Gaudi

DevFeed: [🚀 Accelerating LLM Inference with TGI on Intel Gaudi](<https://devfeed.tech/articles/accelerating-llm-inference-with-tgi-on-intel-gaudi-7289.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/intel-gaudi-backend-for-tgi>)

Author: Baptiste Colle; Régis Pierrard; Ilyas Moutawwakil; Ella Charlaix; Ke Ding

Published: 2025-03-28T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [tgi](<https://devfeed.tech/topics/tgi.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [intel](<https://devfeed.tech/topics/intel.md>), [model-deployment](<https://devfeed.tech/topics/model-deployment.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>)

Tags: [aws](<https://devfeed.tech/tags/aws.md>), [batching](<https://devfeed.tech/tags/batching.md>), [dell](<https://devfeed.tech/tags/dell.md>), [docker](<https://devfeed.tech/tags/docker.md>), [gaudi](<https://devfeed.tech/tags/gaudi.md>), [gemma](<https://devfeed.tech/tags/gemma.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [ibm](<https://devfeed.tech/tags/ibm.md>), [inference](<https://devfeed.tech/tags/inference.md>), [intel](<https://devfeed.tech/tags/intel.md>), [llm](<https://devfeed.tech/tags/llm.md>), [mixtral](<https://devfeed.tech/tags/mixtral.md>), [partnerships](<https://devfeed.tech/tags/partnerships.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [tgi](<https://devfeed.tech/tags/tgi.md>)

### AI overview

This article announces the integration of Intel Gaudi support into the main TGI codebase through TGI's multi-backend architecture. It describes deployment options across Gaudi hardware generations, production features such as dynamic batching and streamed responses, support for multiple LLMs, multi-card inference, vision-language models, FP8 precision, and an official Docker image.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.

## Introducing multi-backends (TRT-LLM, vLLM) support for Text Generation Inference

DevFeed: [Introducing multi-backends (TRT-LLM, vLLM) support for Text Generation Inference](<https://devfeed.tech/articles/introducing-multi-backends-trt-llm-vllm-support-for-text-generation-inference-7501.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/tgi-multi-backend>)

Author: Morgan Funtowicz; Hugo Larcher

Published: 2025-01-16T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [tgi](<https://devfeed.tech/topics/tgi.md>), [model-deployment](<https://devfeed.tech/topics/model-deployment.md>), [Rust](<https://devfeed.tech/topics/rust.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [TensorRT-LLM](<https://devfeed.tech/topics/tensorrt-llm.md>), [Concurrency](<https://devfeed.tech/topics/concurrency.md>), [Memory Safety](<https://devfeed.tech/topics/memory-safety.md>), [servers](<https://devfeed.tech/topics/servers.md>), [llama.cpp](<https://devfeed.tech/topics/llama-cpp.md>), [sglang](<https://devfeed.tech/topics/sglang.md>), [Python](<https://devfeed.tech/topics/python.md>), [Code](<https://devfeed.tech/topics/code.md>)

Tags: [backend](<https://devfeed.tech/tags/backend.md>), [backends](<https://devfeed.tech/tags/backends.md>), [community](<https://devfeed.tech/tags/community.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [http](<https://devfeed.tech/tags/http.md>), [inference](<https://devfeed.tech/tags/inference.md>), [jetstream](<https://devfeed.tech/tags/jetstream.md>), [llama-cpp](<https://devfeed.tech/tags/llama-cpp.md>), [memory-safety](<https://devfeed.tech/tags/memory-safety.md>), [neuron](<https://devfeed.tech/tags/neuron.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [performance](<https://devfeed.tech/tags/performance.md>), [production](<https://devfeed.tech/tags/production.md>), [python](<https://devfeed.tech/tags/python.md>), [rust](<https://devfeed.tech/tags/rust.md>), [sglang](<https://devfeed.tech/tags/sglang.md>), [tensorrt-llm](<https://devfeed.tech/tags/tensorrt-llm.md>), [tgi](<https://devfeed.tech/tags/tgi.md>), [vllm](<https://devfeed.tech/tags/vllm.md>)

### AI overview

Hugging Face introduces TGI Backends, a unified frontend architecture for integrating inference solutions such as vLLM, SGLang, llama.cpp, and TensorRT-LLM. The approach lets users switch backends based on model, hardware, and performance requirements while supporting production deployment across diverse accelerators. The article also describes TGI's Rust and Python components, including Rust-based HTTP and scheduling layers designed for memory safety and concurrency.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.

## Introducing HUGS - Scale your AI with Open Models

DevFeed: [Introducing HUGS - Scale your AI with Open Models](<https://devfeed.tech/articles/introducing-hugs-scale-your-ai-with-open-models-7255.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/hugs>)

Author: Philipp Schmid; Jeff Boudier; Alvaro Bartolome; Simon Pagezy; Violette

Published: 2024-10-23T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [hugging face](<https://devfeed.tech/topics/hugging-face.md>), [model-deployment](<https://devfeed.tech/topics/model-deployment.md>), [tgi](<https://devfeed.tech/topics/tgi.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [Generative AI](<https://devfeed.tech/topics/generative-ai.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [Microservices](<https://devfeed.tech/topics/microservices.md>), [API](<https://devfeed.tech/topics/api.md>), [Hardware](<https://devfeed.tech/topics/hardware.md>)

Tags: [accelerate](<https://devfeed.tech/tags/accelerate.md>), [accelerators](<https://devfeed.tech/tags/accelerators.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-accelerator](<https://devfeed.tech/tags/ai-accelerator.md>), [amd](<https://devfeed.tech/tags/amd.md>), [announcement](<https://devfeed.tech/tags/announcement.md>), [api](<https://devfeed.tech/tags/api.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-inferentia](<https://devfeed.tech/tags/aws-inferentia.md>), [azure](<https://devfeed.tech/tags/azure.md>), [digitalocean](<https://devfeed.tech/tags/digitalocean.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [gcp](<https://devfeed.tech/tags/gcp.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [inference](<https://devfeed.tech/tags/inference.md>), [llm](<https://devfeed.tech/tags/llm.md>), [microservices](<https://devfeed.tech/tags/microservices.md>), [model-deployment](<https://devfeed.tech/tags/model-deployment.md>), [models](<https://devfeed.tech/tags/models.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [partnerships](<https://devfeed.tech/tags/partnerships.md>), [performance](<https://devfeed.tech/tags/performance.md>), [technologies](<https://devfeed.tech/tags/technologies.md>), [text-generation](<https://devfeed.tech/tags/text-generation.md>), [tgi](<https://devfeed.tech/tags/tgi.md>), [transformers](<https://devfeed.tech/tags/transformers.md>)

### AI overview

Hugging Face introduces HUGS, optimized zero-configuration inference microservices for deploying open models in an organization's own infrastructure. Built on Hugging Face technologies including Text Generation Inference and Transformers, HUGS targets efficient, hardware-optimized deployment across NVIDIA and AMD GPUs, with AWS Inferentia and Google TPU support planned. It provides an OpenAI-compatible API and is designed to reduce deployment complexity and time for AI applications.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.

## Llama can now see and run on your device - welcome Llama 3.2

DevFeed: [Llama can now see and run on your device - welcome Llama 3.2](<https://devfeed.tech/articles/llama-can-now-see-and-run-on-your-device-welcome-llama-3-2-7337.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/llama32>)

Author: merve; Philipp Schmid; Omar Sanseviero; Vaibhav Srivastav; Lewis Tunstall; Aritra Roy Gosthipaty; Pedro Cuenca

Published: 2024-09-25T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [llama](<https://devfeed.tech/topics/llama.md>), [multimodal](<https://devfeed.tech/topics/multimodal.md>), [Meta](<https://devfeed.tech/topics/meta.md>), [hugging face](<https://devfeed.tech/topics/hugging-face.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [model-deployment](<https://devfeed.tech/topics/model-deployment.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [inference-endpoints](<https://devfeed.tech/topics/inference-endpoints.md>), [tgi](<https://devfeed.tech/topics/tgi.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>)

Tags: [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [hugging-face](<https://devfeed.tech/tags/hugging-face.md>), [inference](<https://devfeed.tech/tags/inference.md>), [inference-endpoints](<https://devfeed.tech/tags/inference-endpoints.md>), [llama](<https://devfeed.tech/tags/llama.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llms](<https://devfeed.tech/tags/llms.md>), [meta](<https://devfeed.tech/tags/meta.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [nlp](<https://devfeed.tech/tags/nlp.md>), [on-device](<https://devfeed.tech/tags/on-device.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [tgi](<https://devfeed.tech/tags/tgi.md>), [transformers](<https://devfeed.tech/tags/transformers.md>), [vision](<https://devfeed.tech/tags/vision.md>)

### AI overview

Meta's Llama 3.2 release introduces multimodal Vision models in 11B and 90B sizes, smaller text-only 1B and 3B models for on-device use, and vision-enabled Llama Guard 3. The article describes their capabilities, architecture, supported languages, inference examples, and integrations with Hugging Face Transformers, TGI, Inference Endpoints, Google Cloud, Amazon SageMaker, and DELL Enterprise Hub.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.

## Deploy Meta Llama 3.1 405B on Google Cloud Vertex AI

DevFeed: [Deploy Meta Llama 3.1 405B on Google Cloud Vertex AI](<https://devfeed.tech/articles/deploy-meta-llama-3-1-405b-on-google-cloud-vertex-ai-7336.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/llama31-on-vertex-ai>)

Author: Alvaro Bartolome; Philipp Schmid; Simon Pagezy; Jeff Boudier

Published: 2024-08-19T00:00:00Z

Content type: tutorial

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [llama](<https://devfeed.tech/topics/llama.md>), [Google Cloud Platform (GCP)](<https://devfeed.tech/topics/google-cloud.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [tgi](<https://devfeed.tech/topics/tgi.md>), [Meta](<https://devfeed.tech/topics/meta.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [hugging face](<https://devfeed.tech/topics/hugging-face.md>), [Containers](<https://devfeed.tech/topics/containers.md>)

Tags: [container](<https://devfeed.tech/tags/container.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [gcp](<https://devfeed.tech/tags/gcp.md>), [google-cloud](<https://devfeed.tech/tags/google-cloud.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [hugging-face](<https://devfeed.tech/tags/hugging-face.md>), [inference](<https://devfeed.tech/tags/inference.md>), [llama](<https://devfeed.tech/tags/llama.md>), [nlp](<https://devfeed.tech/tags/nlp.md>), [partnerships](<https://devfeed.tech/tags/partnerships.md>), [vertex](<https://devfeed.tech/tags/vertex.md>), [vertex-ai](<https://devfeed.tech/tags/vertex-ai.md>)

### AI overview

This tutorial explains how to programmatically deploy the FP8-quantized Meta Llama 3.1 405B model on Google Cloud Vertex AI using Text Generation Inference and Hugging Face Deep Learning Containers. It covers deployment on an A3 node with eight NVIDIA H100 GPUs, alternative deployment paths, and the memory considerations for running the model.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.

## TGI Multi-LoRA: Deploy Once, Serve 30 Models

DevFeed: [TGI Multi-LoRA: Deploy Once, Serve 30 Models](<https://devfeed.tech/articles/tgi-multi-lora-deploy-once-serve-30-models-7359.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/multi-lora-serving>)

Author: Derek Thomas; Diego Maniloff; David Holtz

Published: 2024-07-18T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [lora](<https://devfeed.tech/topics/lora.md>), [tgi](<https://devfeed.tech/topics/tgi.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Deployment](<https://devfeed.tech/topics/deployment.md>), [AI Inference](<https://devfeed.tech/topics/ai-inference.md>)

Tags: [cost](<https://devfeed.tech/tags/cost.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [guide](<https://devfeed.tech/tags/guide.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [llm](<https://devfeed.tech/tags/llm.md>), [lora](<https://devfeed.tech/tags/lora.md>), [nlp](<https://devfeed.tech/tags/nlp.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [peft](<https://devfeed.tech/tags/peft.md>), [performance](<https://devfeed.tech/tags/performance.md>), [privacy](<https://devfeed.tech/tags/privacy.md>), [tgi](<https://devfeed.tech/tags/tgi.md>)

### AI overview

This article introduces TGI Multi-LoRA serving, a feature that allows organizations to deploy one base model and serve many specialized models. It explains how LoRA efficiently fine-tunes large pre-trained models by adding small adapter parameter sets, reducing storage and memory overhead while preserving model quality. The feature addresses the cost and operational complexity of deploying multiple fine-tuned Large Language Models.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.

## Google Cloud TPUs made available to Hugging Face users

DevFeed: [Google Cloud TPUs made available to Hugging Face users](<https://devfeed.tech/articles/google-cloud-tpus-made-available-to-hugging-face-users-7524.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/tpu-inference-endpoints-spaces>)

Author: Simon Pagezy; Michelle Habonneau; Philipp Schmid; Alvaro Moran

Published: 2024-07-09T00:00:00Z

Content type: release

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [inference-endpoints](<https://devfeed.tech/topics/inference-endpoints.md>), [Sovereign AI](<https://devfeed.tech/topics/sovereign-ai.md>), [Google Cloud Platform (GCP)](<https://devfeed.tech/topics/google-cloud.md>), [hugging face](<https://devfeed.tech/topics/hugging-face.md>), [spaces](<https://devfeed.tech/topics/spaces.md>), [optimum](<https://devfeed.tech/topics/optimum.md>), [tgi](<https://devfeed.tech/topics/tgi.md>), [gemma](<https://devfeed.tech/topics/gemma.md>), [llama](<https://devfeed.tech/topics/llama.md>), [model-deployment](<https://devfeed.tech/topics/model-deployment.md>), [Latency](<https://devfeed.tech/topics/latency.md>), [Hardware](<https://devfeed.tech/topics/hardware.md>)

Tags: [gcp](<https://devfeed.tech/tags/gcp.md>), [gemma](<https://devfeed.tech/tags/gemma.md>), [google-cloud](<https://devfeed.tech/tags/google-cloud.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [hugging-face](<https://devfeed.tech/tags/hugging-face.md>), [inference](<https://devfeed.tech/tags/inference.md>), [inference-endpoints](<https://devfeed.tech/tags/inference-endpoints.md>), [latency](<https://devfeed.tech/tags/latency.md>), [llama](<https://devfeed.tech/tags/llama.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [optimum](<https://devfeed.tech/tags/optimum.md>), [partnerships](<https://devfeed.tech/tags/partnerships.md>), [performance](<https://devfeed.tech/tags/performance.md>), [spaces](<https://devfeed.tech/tags/spaces.md>), [tgi](<https://devfeed.tech/tags/tgi.md>), [tpu](<https://devfeed.tech/tags/tpu.md>)

### AI overview

Hugging Face announces that Google Cloud TPUs are available for Inference Endpoints and Spaces. Google TPU v5e configurations can deploy supported models through managed infrastructure, while Optimum TPU and Text Generation Inference help train and serve models on TPUs.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.

## Benchmarking Text Generation Inference

DevFeed: [Benchmarking Text Generation Inference](<https://devfeed.tech/articles/benchmarking-text-generation-inference-7500.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/tgi-benchmarking>)

Author: Derek Thomas

Published: 2024-05-29T00:00:00Z

Content type: tutorial

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [benchmarking](<https://devfeed.tech/topics/benchmarking.md>), [tgi](<https://devfeed.tech/topics/tgi.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [model-deployment](<https://devfeed.tech/topics/model-deployment.md>), [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>)

Tags: [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [guide](<https://devfeed.tech/tags/guide.md>), [inference](<https://devfeed.tech/tags/inference.md>), [llm](<https://devfeed.tech/tags/llm.md>), [nlp](<https://devfeed.tech/tags/nlp.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [performance](<https://devfeed.tech/tags/performance.md>), [rag](<https://devfeed.tech/tags/rag.md>), [techniques](<https://devfeed.tech/tags/techniques.md>), [tgi](<https://devfeed.tech/tags/tgi.md>), [watermarking](<https://devfeed.tech/tags/watermarking.md>)

### AI overview

This article explains how to benchmark and profile Text Generation Inference (TGI) for different large language model workloads and hardware configurations. It discusses serving optimizations and contrasts the context-window demands of retrieval-augmented generation with basic chat.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.

## Deploy models on AWS Inferentia2 from Hugging Face

DevFeed: [Deploy models on AWS Inferentia2 from Hugging Face](<https://devfeed.tech/articles/deploy-models-on-aws-inferentia2-from-hugging-face-7285.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/inferentia-inference-endpoints>)

Author: Jeff Boudier; Philipp Schmid

Published: 2024-05-22T00:00:00Z

Content type: release

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [AWS AI chips](<https://devfeed.tech/topics/aws-ai-chips.md>), [inference-endpoints](<https://devfeed.tech/topics/inference-endpoints.md>), [hugging face](<https://devfeed.tech/topics/hugging-face.md>), [Amazon SageMaker](<https://devfeed.tech/topics/amazon-sagemaker.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [tgi](<https://devfeed.tech/topics/tgi.md>), [autoscaling](<https://devfeed.tech/topics/autoscaling.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [MLOps](<https://devfeed.tech/topics/mlops.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>)

Tags: [autoscaling](<https://devfeed.tech/tags/autoscaling.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-trainium](<https://devfeed.tech/tags/aws-trainium.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [hugging-face](<https://devfeed.tech/tags/hugging-face.md>), [inference](<https://devfeed.tech/tags/inference.md>), [inference-endpoints](<https://devfeed.tech/tags/inference-endpoints.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [llama3](<https://devfeed.tech/tags/llama3.md>), [mlops](<https://devfeed.tech/tags/mlops.md>), [model](<https://devfeed.tech/tags/model.md>), [optimum](<https://devfeed.tech/tags/optimum.md>), [partnership](<https://devfeed.tech/tags/partnership.md>), [tgi](<https://devfeed.tech/tags/tgi.md>)

### AI overview

Hugging Face announces support for deploying models on AWS Inferentia2 through Amazon SageMaker and Hugging Face Inference Endpoints. The update enables scalable inference for supported models, including Meta Llama 3, with multiple instance sizes, managed features, and autoscaling.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.

## Hugging Face on AMD Instinct MI300 GPU

DevFeed: [Hugging Face on AMD Instinct MI300 GPU](<https://devfeed.tech/articles/hugging-face-on-amd-instinct-mi300-gpu-7245.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/huggingface-amd-mi300>)

Author: Félix Marty; Mohit Sharma; seungrok jung; Morgan Funtowicz

Published: 2024-05-21T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [hugging face](<https://devfeed.tech/topics/hugging-face.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [AI Models](<https://devfeed.tech/topics/ai-models.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>), [tgi](<https://devfeed.tech/topics/tgi.md>), [Azure](<https://devfeed.tech/topics/azure.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [Kubernetes](<https://devfeed.tech/topics/kubernetes.md>), [text-generation](<https://devfeed.tech/topics/text-generation.md>), [migration](<https://devfeed.tech/topics/migration.md>)

Tags: [ai-models](<https://devfeed.tech/tags/ai-models.md>), [amd](<https://devfeed.tech/tags/amd.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [azure](<https://devfeed.tech/tags/azure.md>), [ci-cd](<https://devfeed.tech/tags/ci-cd.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [hugging-face](<https://devfeed.tech/tags/hugging-face.md>), [inference](<https://devfeed.tech/tags/inference.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [instinct](<https://devfeed.tech/tags/instinct.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [llama](<https://devfeed.tech/tags/llama.md>), [llm](<https://devfeed.tech/tags/llm.md>), [migration](<https://devfeed.tech/tags/migration.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [optimum](<https://devfeed.tech/tags/optimum.md>), [performance](<https://devfeed.tech/tags/performance.md>), [rocm](<https://devfeed.tech/tags/rocm.md>), [text-generation](<https://devfeed.tech/tags/text-generation.md>), [transformers](<https://devfeed.tech/tags/transformers.md>)

### AI overview

Hugging Face and AMD describe first-class integration for AMD Instinct MI300 GPU servers across the Hugging Face Platform. The article covers deployment from local development to Azure ND MI300x V5 VMs, compatibility with existing libraries and products, and CI/CD testing on managed Kubernetes infrastructure.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.

## Ryght's Journey to Empower Healthcare and Life Sciences with Expert Support from Hugging Face

DevFeed: [Ryght's Journey to Empower Healthcare and Life Sciences with Expert Support from Hugging Face](<https://devfeed.tech/articles/ryght-s-journey-to-empower-healthcare-and-life-sciences-with-expert-support-from-hugging-face-7461.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/ryght-case-study>)

Author: Andrew Reed; Johnny Crupi

Published: 2024-04-16T00:00:00Z

Content type: article

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [hugging face](<https://devfeed.tech/topics/hugging-face.md>), [AI Platform](<https://devfeed.tech/topics/ai-platform.md>), [Generative AI](<https://devfeed.tech/topics/generative-ai.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [tgi](<https://devfeed.tech/topics/tgi.md>), [Software as a service](<https://devfeed.tech/topics/saas.md>), [Embeddings](<https://devfeed.tech/topics/embeddings.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>)

Tags: [ai-platform](<https://devfeed.tech/tags/ai-platform.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [case-studies](<https://devfeed.tech/tags/case-studies.md>), [embeddings](<https://devfeed.tech/tags/embeddings.md>), [expert-support](<https://devfeed.tech/tags/expert-support.md>), [expert-support-program](<https://devfeed.tech/tags/expert-support-program.md>), [generative-ai](<https://devfeed.tech/tags/generative-ai.md>), [healthcare](<https://devfeed.tech/tags/healthcare.md>), [hugging-face](<https://devfeed.tech/tags/hugging-face.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [partnership](<https://devfeed.tech/tags/partnership.md>), [saas](<https://devfeed.tech/tags/saas.md>), [tgi](<https://devfeed.tech/tags/tgi.md>)

### AI overview

Ryght launched Ryght Preview, an enterprise-grade generative AI SaaS platform for healthcare and life sciences. The platform provides industry-specific AI copilots and custom solutions for analyzing and documenting complex data sources. Hugging Face supports the company through its Expert Support Program, including advisory meetings, technical workshops, and access to Text Generation Inference (TGI) and Text Embeddings Inference (TEI) services.

### Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.