# llama.cpp llama.cpp is an open-source C/C++ implementation for running large language and vision-language model inference locally or in the cloud across diverse hardware. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Sampling args in llama-server DevFeed: [Sampling args in llama-server]() Original publisher: [Read original article]() Author: Alex Ewerlöf Published: 2026-07-01T18:28:35Z Content type: tutorial Language: en Sources: [Alex Ewerlof Notes]() Topics: [llama.cpp](), [Large Language Model](), [Inference](), [quantization](), [Benchmark]() Tags: [benchmark](), [hallucinations](), [inference](), [llama-cpp](), [llm](), [quantization]() ### AI overview A reference guide to llama.cpp sampling and repetition parameters, covering common failure modes in local and quantized language models, parameter ranges and defaults, workload-specific tuning, modern sampling methods, and ways to accelerate experimentation. ### Source excerpt Reducing repetition, hallucinations, degradation, while making inference faster! ## Using local LLMs for agentic coding DevFeed: [Using local LLMs for agentic coding]() Original publisher: [Read original article]() Author: Alex Ewerlöf Published: 2026-06-04T09:01:34Z Content type: tutorial Language: en Sources: [Alex Ewerlof Notes]() Topics: [coding](), [agentic-coding](), [Large Language Model](), [GitHub Copilot CLI](), [Linux](), [llama.cpp](), [Ollama](), [NVIDIA RTX]() Tags: [agentic-coding](), [agents](), [ai](), [claude](), [coding](), [github](), [linux](), [llama](), [llama-cpp](), [llms](), [local](), [local-llms](), [microsoft](), [nvidia](), [nvidia-rtx](), [ollama](), [quality](), [rocm](), [state]() ### AI overview A practical guide to using local language models for agentic coding. It covers running local models, configuring coding agents such as Copilot and Pi, and evaluating local models against cloud models in terms of cost, privacy, tooling, and performance. ### Source excerpt AI honeymoon pricing is over, but your work is not ## Use Your Mac for AI Agents: Self-Host Gemma 4 12 B with Pulumi and Tailscale DevFeed: [Use Your Mac for AI Agents: Self-Host Gemma 4 12 B with Pulumi and Tailscale]() Original publisher: [Read original article]() Author: Pablo Seibelt Published: 2026-06-04T00:00:00Z Content type: tutorial Language: en Sources: [Pulumi]() Topics: [gemma4](), [llama.cpp](), [Local AI](), [Infrastructure as code](), [macOS](), [On-device AI](), [multimodal]() Tags: [ai](), [download](), [gemma-4](), [gemma4](), [gpu](), [infrastructure-as-code](), [kubernetes](), [llama-cpp](), [macos](), [multimodal](), [python](), [tailscale](), [tutorials]() ### AI overview A tutorial for self-hosting Gemma 4 12 B on a modern Mac using llama.cpp with Apple Metal acceleration. It combines host-native inference with a local Kubernetes cluster, Pulumi infrastructure as code, and Tailscale for secure access, and reports validation results on a MacBook Pro with an Apple M3 Max and 36 GB RAM. ### Source excerpt If you run AI tools and agents, you've probably accepted three tradeoffs: your data leaves your network, you can't work offline, and your bill scales with usage. Open-weight models now run well on consumer hardware. Once the model is on your machine, your data stays local, inference works offline, and tokens cost nothing. If you own a modern Mac, you can run a high-quality model yourself. Gemma 4 is an open-weights model family from Google. This post focuses on Gemma 4 12 B, released in June 2026, using Unsloth's Q8_0 GGUF. The 12 B model fits comfortably on a modern Mac while leaving enough headroom for local llama.cpp and a chat UI. We'll use llama.cpp for host-native inference, k3d for a local Kubernetes cluster, Pulumi for infrastructure as code, and Tailscale for secure access. Prerequisites This setup was validated on the following hardware: macOS 26 Tahoe, version 26.5 MacBook Pro with Apple M3 Max 36 GB RAM On this machine, llama.cpp reported about 20 output tokens per second for a 160-token validation response with unsloth/gemma-4-12b-it-GGUF, gemma-4-12b-it-Q8_0.gguf, and a 131,072-token context. Sustained throughput varies by prompt length, thermal state, and llama.cpp settings. You'll need brew, docker, pulumi, and tailscale installed. We'll also install k3d during the process. Run Gemma 4 with host-native llama.cpp We use llama.cpp directly on macOS to leverage Apple Metal acceleration. Running the LLM on the host is more efficient than trying to pass GPU access into a local Kubernetes VM. Install the build tools: brew install cmake git Then build llama.cpp from source and download the multimodal projector. In validation, Homebrew llama.cpp 9430 could run text inference, but it could not load the new Gemma 4 12 B projector and failed with unknown projector type: gemma4uv. Building current llama.cpp from source fixed that. llm_home="$HOME/pulumi-gemma4-llm" mkdir -p "$llm_home/models" "$llm_home/logs" if [ ! -d "$llm_home/llama.cpp/.git" ]; then git clon ## WMMA guide for AMD RDNA 4 architecture GPUs - part 3 DevFeed: [WMMA guide for AMD RDNA 4 architecture GPUs - part 3]() Original publisher: [Read original article]() Author: Hui Zhang Published: 2026-06-02T19:02:00Z Content type: tutorial Language: en Sources: [AMD GPUOpen]() Topics: [Code](), [Optimization](), [llama.cpp](), [Warp]() Tags: [3](), [amd](), [architecture](), [code](), [cpp](), [cuda](), [getting-started](), [guide](), [hip](), [implement](), [llama-cpp](), [maths](), [memory](), [ml](), [optimization](), [performance](), [rdna](), [reference-guide](), [technical-article](), [technical-articles](), [user-guides-manuals]() ### AI overview This tutorial explains how to implement an in-register matrix transpose using the WMMA layout on AMD RDNA 4 GPUs. It describes an identity-matrix approach that avoids additional memory operations and notes its use in Llama.cpp for Flash Attention. ### Source excerpt Learn how to implement fast in-register matrix transpose on AMD RDNA™ 4 architecture GPUs with a WMMA-based identity trick, delivering a lightweight, memory-free alternative proven in Llama.cpp. ## How I Run My Daily Standup on a Local LLM: Git, Gemma 4, llama.cpp and Telegram DevFeed: [How I Run My Daily Standup on a Local LLM: Git, Gemma 4, llama.cpp and Telegram]() Original publisher: [Read original article]() Author: Ivan Morgillo Published: 2026-05-27T09:51:06Z Content type: tutorial Language: en Sources: [Ivan Morgillo]() Topics: [gemma4](), [llama.cpp](), [Git](), [Large Language Model](), [API](), [Claude]() Tags: [anthropic](), [api](), [claude](), [gemma](), [gemma-4](), [git](), [llama-cpp](), [llm](), [local](), [local-llm](), [sdk](), [telegram]() ### AI overview A developer describes replacing a cloud-based daily standup pipeline with a local Gemma 4 E4B model running on llama.cpp. The pipeline reads Git commits, formats them, and posts the summary to Telegram. The article also explains how empty API-key settings caused several automations to use personal Claude subscription authentication instead of the Anthropic API. ### Source excerpt A real-world pipeline that scans my Git repos every morning, formats the day's commits with a local Gemma 4 E4B on llama.cpp, and posts the summary to Telegram -- no cloud, no API key, no AUP warnings. ## Reachy Mini goes fully local DevFeed: [Reachy Mini goes fully local]() Original publisher: [Read original article]() Author: Amir Mahla; Andres Marafioti Published: 2026-05-27T00:00:00Z Content type: tutorial Language: en Sources: [Hugging Face - Blog]() Topics: [reachy](), [gemma4](), [llama.cpp](), [Large Language Model](), [speech-to-speech](), [hugging face](), [WebSocket]() Tags: [api](), [artificial-intelligence](), [audio](), [blog](), [gemma](), [guide](), [how-to](), [llama](), [llama-cpp](), [llm](), [local](), [local-ai](), [open-source](), [reachy](), [robotics](), [server](), [speech](), [speech-to-speech]() ### AI overview A tutorial for running fully local conversations with a Reachy Mini robot. It describes a cascaded VAD, speech-to-text, LLM, and text-to-speech pipeline using llama.cpp with Gemma 4, Silero VAD, Parakeet-TDT STT, and Qwen3-TTS, connected through a Realtime API-compatible WebSocket. The setup avoids cloud services, API keys, and sending data off the local machine. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Why and How to Run Local Models in Zed DevFeed: [Why and How to Run Local Models in Zed]() Original publisher: [Read original article]() Author: Cameron Mcloughlin Published: 2026-05-19T00:00:00Z Content type: tutorial Language: en Sources: [Zed Industries - Blog]() Topics: [Local AI](), [LLMs](), [llama.cpp](), [Ollama](), [data]() Tags: [developer](), [hardware](), [llama](), [llama-cpp](), [llms](), [local](), [local-ai](), [ollama]() ### AI overview This article explains why developers may choose local models in Zed, focusing on privacy, cost, control, and availability. It also discusses their limitations compared with cloud-hosted frontier models and introduces setup guidance. ### Source excerpt You can run local AI models in Zed to get better performance and control over your data. Here's how. ## Welcome Gemma 4: Frontier multimodal intelligence on device DevFeed: [Welcome Gemma 4: Frontier multimodal intelligence on device]() Original publisher: [Read original article]() Author: merve; Pedro Cuenca; Sergio Paniego; ben burtenshaw; Steven Zheng; Alvaro Bartolome; Nathan Habib Published: 2026-04-02T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [gemma4](), [multimodal](), [Open Source](), [long-context](), [quantization](), [llama.cpp](), [MLX](), [Rust](), [webgpu]() Tags: [audio](), [benchmarks](), [fine-tuning](), [gemma](), [gemma4](), [generation](), [google](), [guide](), [llama](), [llama-cpp](), [local](), [long-context](), [mlx](), [models](), [moe](), [multimodal](), [on-device](), [open](), [open-source](), [quantization](), [science](), [speed]() ### AI overview Gemma 4 is presented as an open, Apache 2-licensed family of multimodal models supporting image, video, text, and, for selected variants, audio inputs. The article highlights on-device deployment, long-context support, compatibility across libraries and devices, quantization, benchmark results, and implementation options including llama.cpp, MLX, WebGPU, and Rust. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Liberate your OpenClaw DevFeed: [Liberate your OpenClaw]() Original publisher: [Read original article]() Author: Clem 🤗; ben burtenshaw; Pedro Cuenca; Jeff Boudier; merve; Niels Rogge; Victor Mustar; Mishig ᠮᠢᠰᠾᠢᠭ Published: 2026-03-27T00:00:00Z Content type: tutorial Language: en Sources: [Hugging Face - Blog]() Topics: [OpenClaw](), [Open Source](), [hugging face](), [inference-providers](), [llama.cpp]() Tags: [agents](), [guide](), [hugging-face](), [inference-providers](), [llama-cpp](), [local-ai](), [open-source](), [openclaw](), [privacy](), [server]() ### AI overview This tutorial explains how to restore OpenClaw agents using open models through Hugging Face Inference Providers or by running models locally with llama.cpp. It compares hosted and local approaches, covering privacy, cost, hardware, model selection, configuration, and local server setup. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Local LLM Inference : llama.cpp, GGUF, Quantizations and GGML Explained DevFeed: [Local LLM Inference : llama.cpp, GGUF, Quantizations and GGML Explained]() Original publisher: [Read original article]() Author: Alex Razvant Published: 2026-03-03T11:31:04Z Content type: tutorial Language: en Sources: [Neural Bits]() Topics: [llama.cpp](), [ggml](), [Inference](), [Large Language Model](), [Local AI](), [quantization]() Tags: [backend](), [cross-platform](), [efficiently](), [embedded](), [format](), [ggml](), [inference](), [llama-cpp](), [llm](), [local-llm](), [model]() ### AI overview A practical guide to local LLM inference with llama.cpp, explaining how the GGUF model format, GGML backend concepts, quantization, and inference workflows fit together for efficient execution on edge devices. ### Source excerpt Learn how the llama.cpp runtime, GGML backend concepts, and GGUF model format fit together for fast local inference across devices. ## GTX 1080 Ti for Local LLM DevFeed: [GTX 1080 Ti for Local LLM]() Original publisher: [Read original article]() Published: 2026-02-28T23:33:54Z Content type: article Language: en Sources: [Ariya Hidayat]() Topics: [GPU](), [LLMs](), [llama.cpp](), [Nvidia](), [CUDA](), [Inference Performance](), [Retrieval Augmented Generation (RAG)]() Tags: [cmake](), [command-line](), [cuda](), [gcc](), [gemma](), [gpu](), [inference](), [inference-performance](), [llama](), [llama-cpp](), [llms](), [nvidia]() ### AI overview This article evaluates the eight-year-old NVIDIA GTX 1080 Ti as an affordable option for running local LLMs. It explains how its 11 GB of VRAM can support quantized models and relevant context windows, then describes measuring prompt-processing and token-generation performance with CUDA-enabled llama.cpp. ### Source excerpt Despite being over eight years old, the NVIDIA GTX 1080 Ti remains a compelling choice for enthusiasts keen on running LLM locally. ## GGML and llama.cpp join HF to ensure the long-term progress of Local AI DevFeed: [GGML and llama.cpp join HF to ensure the long-term progress of Local AI]() Original publisher: [Read original article]() Author: Georgi Gerganov; Xuan-Son Nguyen; Aleksander Grygier; Lysandre; Victor Mustar; Julien Chaumond Published: 2026-02-20T00:00:00Z Content type: news Language: en Sources: [Hugging Face - Blog]() Topics: [llama.cpp](), [Local AI](), [Inference](), [Open Source](), [model-deployment](), [Artificial Intelligence](), [User Experience]() Tags: [community](), [devices](), [ggml](), [inference](), [llama-cpp](), [llm](), [local-ai](), [open-source](), [superintelligence](), [user-experience]() ### AI overview Georgi Gerganov and the ggml team are joining Hugging Face to support the llama.cpp and ggml communities while retaining autonomy over technical direction and community leadership. The collaboration will provide long-term resources, improve integration between model definitions and llama.cpp, simplify packaging and user experience, and expand access to efficient local inference on devices. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Granite 4.0 Nano: Just how small can you go? DevFeed: [Granite 4.0 Nano: Just how small can you go?]() Original publisher: [Read original article]() Author: Kate Soule; Rameswar Panda Published: 2025-10-28T14:59:38Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [ibm](), [Large Language Model](), [Model Development](), [Open Source Models & Datasets](), [llama.cpp](), [MLX](), [vllm](), [Transformer](), [Loop Engineering](), [gemma](), [qwen](), [Google]() Tags: [apache](), [architecture](), [benchmarks](), [code](), [data](), [development](), [gemma](), [google](), [ibm](), [language-models](), [llama](), [llama-cpp](), [math](), [mlx](), [model](), [model-development](), [models](), [on-device](), [qwen](), [training]() ### AI overview IBM introduces Granite 4.0 Nano, a family of compact language models for edge and on-device applications. The release includes hybrid-SSM and traditional transformer variants ranging from roughly 350M to 1.5B parameters, supports vLLM, llama.cpp, and MLX, and is released under the Apache 2.0 license. The article reports strong performance across knowledge, math, code, safety, instruction-following, and tool-calling benchmarks. ### Source excerpt Today we are excited to share Granite 4.0 Nano, our smallest models yet, released as part of IBM's Granite 4.0 model family. Designed for the edge and on-device applications, these models demonstrate excellent performance for their size and represent IBM's continued commitment to develop powerful, useful, models that don't require hundreds of billions of parameters to get the job done. ## Welcome GPT OSS, the new open-source model family from OpenAI! DevFeed: [Welcome GPT OSS, the new open-source model family from OpenAI!]() Original publisher: [Read original article]() Author: Vaibhav Srivastav; Pedro Cuenca; Lewis Tunstall; Clem 🤗; Matthew Carrigan; Clémentine Fourrier; Célina Hanouti; Lucain Pouget; Marc Sun; Simon Pagezy Published: 2025-08-05T00:00:00Z Content type: release Language: en Sources: [Hugging Face - Blog]() Topics: [gpt-oss](), [Open Source](), [OpenAI](), [Artificial Intelligence](), [inference-providers](), [llama.cpp](), [moe](), [Ollama](), [Python](), [quantization](), [Transformers](), [vllm]() Tags: [api](), [artificial-intelligence](), [community](), [gpt](), [gpt-oss](), [inference-providers](), [javascript](), [llama-cpp](), [llm](), [moe](), [ollama](), [open-source](), [openai](), [python](), [quantization](), [transformers](), [vllm]() ### AI overview Hugging Face welcomes OpenAI's gpt-oss open-source model family. The article describes the models' Apache 2.0 licensing, local deployment options, reasoning and tool-use capabilities, MoE architecture, quantization, supported inference implementations, and access through Inference Providers and the Responses API. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Gemma 3n fully available in the open-source ecosystem! DevFeed: [Gemma 3n fully available in the open-source ecosystem!]() Original publisher: [Read original article]() Author: Aritra Roy Gosthipaty; Pedro Cuenca; Sergio Paniego; Vaibhav Srivastav; Christopher Fleetwood; Joshua; Steven Zheng; Kashif Rasul Published: 2025-06-26T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [gemma](), [Open Source](), [AI Development](), [llama.cpp](), [MLX](), [Ollama](), [transformers.js](), [GPU](), [Hardware](), [timm](), [asr](), [Embeddings]() Tags: [ai](), [artificial-intelligence](), [audio](), [community](), [embeddings](), [gemma](), [gpu](), [hardware](), [implementation](), [llama-cpp](), [llm](), [mlx](), [multimodal](), [ollama](), [open-source](), [release](), [research](), [speech](), [timm](), [transformers](), [transformers-js](), [translation](), [vision](), [vlm]() ### AI overview This article announces the availability of Gemma 3n in major open-source libraries and presents practical usage and fine-tuning examples. It describes the model variants, memory-efficient hardware requirements, multimodal audio and vision encoders, speech-to-text and translation capabilities, and architectural features including MatFormer and Per-Layer Embeddings. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Structured Output with LangChain and Llamafile DevFeed: [Structured Output with LangChain and Llamafile]() Original publisher: [Read original article]() Author: brakmic Published: 2025-06-22T16:50:12Z Content type: tutorial Language: en Sources: [Harris Brakmic - Coding]() Topics: [LangChain](), [llamafile](), [JSON](), [llama.cpp](), [Parser](), [Pydantic]() Tags: [ai](), [code](), [coding](), [cpp](), [how-to](), [import](), [json](), [langchain](), [llama-cpp](), [llamafile](), [llm](), [local](), [local-llm](), [models](), [properties](), [python](), [structured-output]() ### AI overview A tutorial on using LangChain with Llamafile to produce structured JSON output from a local large language model. It uses JsonOutputParser, PromptTemplate, and a Pydantic model because Llamafile does not provide a with_structured_output method. ### Source excerpt Learn how to extend Llamafile with LangChain's JsonParser to produce clean, structured JSON output ## The Transformers Library: standardizing model definitions DevFeed: [The Transformers Library: standardizing model definitions]() Original publisher: [Read original article]() Author: Lysandre; Arthur Zucker; Pedro Cuenca; Julien Chaumond Published: 2025-05-15T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [Transformer](), [LLMs](), [interoperability](), [sglang](), [vllm](), [llama.cpp](), [tgi](), [MLX]() Tags: [announcement](), [community](), [interoperability](), [library](), [llama-cpp](), [llms](), [mlx](), [open-source](), [sglang](), [tgi](), [transformers](), [vllm](), [vlms]() ### AI overview The article presents Transformers as a standard library for defining and supporting machine learning model architectures. It describes its broad ecosystem integrations, including training frameworks and inference engines, and highlights interoperability with vLLM, SGLang, TGI, llama.cpp, and MLX. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Run Large Language Models Locally on a Mac with Ollama DevFeed: [Run Large Language Models Locally on a Mac with Ollama]() Original publisher: [Read original article]() Author: Mattt (mattt@nshipster.com) Published: 2025-02-14T08:00:00Z Content type: tutorial Language: en Sources: [NSHipster]() Topics: [Ollama](), [Large Language Model](), [llama.cpp](), [llama3](), [Containers](), [Homebrew]() Tags: [artificial-intelligence](), [containers](), [homebrew](), [inference](), [latency](), [llama-cpp](), [llama3](), [llms](), [miscellaneous](), [oci](), [ollama]() ### AI overview This tutorial explains how to use Ollama to download, run, and manage large language models locally on a Mac. It covers installation, llama3.2, llama.cpp, Modelfiles, OCI-based model distribution, and the practical benefits of local inference, including privacy, cost, latency, control, and reliability. ### Source excerpt While we wait for Apple Intelligence to arrive on our devices, something remarkable is already running on our Macs. Think of it as a locavore approach to artificial intelligence: homegrown, sustainable, and available year-round. ## Introducing multi-backends (TRT-LLM, vLLM) support for Text Generation Inference DevFeed: [Introducing multi-backends (TRT-LLM, vLLM) support for Text Generation Inference]() Original publisher: [Read original article]() Author: Morgan Funtowicz; Hugo Larcher Published: 2025-01-16T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [tgi](), [model-deployment](), [Rust](), [vllm](), [TensorRT-LLM](), [Concurrency](), [Memory Safety](), [servers](), [llama.cpp](), [sglang](), [Python](), [Code]() Tags: [backend](), [backends](), [community](), [concurrency](), [http](), [inference](), [jetstream](), [llama-cpp](), [memory-safety](), [neuron](), [open-source](), [performance](), [production](), [python](), [rust](), [sglang](), [tensorrt-llm](), [tgi](), [vllm]() ### AI overview Hugging Face introduces TGI Backends, a unified frontend architecture for integrating inference solutions such as vLLM, SGLang, llama.cpp, and TensorRT-LLM. The approach lets users switch backends based on model, hardware, and performance requirements while supporting production deployment across diverse accelerators. The article also describes TGI's Rust and Python components, including Rust-based HTTP and scheduling layers designed for memory safety and concurrency. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Hello Deep Learning: Further reading & worthwhile projects DevFeed: [Hello Deep Learning: Further reading & worthwhile projects]() Original publisher: [Read original article]() Published: 2023-03-30T10:00:09Z Content type: article Language: en Sources: [Bert Hubert's writings]() Topics: [Deep learning](), [PyTorch](), [Code](), [C++](), [llama.cpp](), [Whisper](), [llama](), [Open Source](), [jupyter notebooks]() Tags: [c-plus-plus](), [cpp](), [deep-learning](), [open-source](), [projects](), [python](), [pytorch](), [reading]() ### AI overview This article concludes the Hello Deep Learning series with a short list of further-reading resources and projects. It recommends PyTorch documentation, educational material by Andrej Karpathy, FastAI Jupyter notebooks, and local open-source projects including Whisper.cpp and Llama.cpp. ### Source excerpt This page is part of the Hello Deep Learning series of blog posts. You are very welcome to improve this page via GitHub! After having completed this series of blogposts (well done!) you should have a good grounding in what deep learning is actually doing. However, this was of course only a small 20k word introduction, so there is a lot left to learn. Unfortunately, there is a lot of nonsense online.