# llama.cpp Published articles for llama.cpp. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## MLPerf Inference v6.1: 5.7x Per-Accelerator Gains, a 512-GPU Run, and Vera Rubin's First Peer-Reviewed Numbers DevFeed: [MLPerf Inference v6.1: 5.7x Per-Accelerator Gains, a 512-GPU Run, and Vera Rubin's First Peer-Reviewed Numbers]() Original publisher: [Read original article]() Author: Harold Fritts Published: 2026-09-16T15:00:00Z Content type: news Language: en Sources: [StorageReview.com]() Topics: [Benchmark](), [Inference Performance](), [Retrieval Augmented Generation (RAG)](), [agentic-coding](), [GPU](), [Vera Rubin NVL72](), [Nvidia](), [Vera Rubin]() Tags: [agentic-coding](), [ai](), [benchmark](), [enterprise](), [gpt-oss](), [gpu](), [inference](), [llama-cpp](), [numbers](), [nvidia](), [qwen3](), [rag]() ### AI overview MLCommons published MLPerf Inference v6.1 with record participation, two new inference tests, and peer-reviewed results for several newly covered accelerators. The release reports a 5.7x improvement in the best per-accelerator DeepSeek-R1 server result compared with v5.1. ### Source excerpt MLCommons has published MLPerf Inference v6.1, and the round sets a participation record with 30 submitting organizations and 486 datacenter and edge results. Two new tests join the suite: an End-to-End RAG pipeline for the datacenter and an Edge Agentic Inference benchmark for single-user devices, and the results carry the first peer-reviewed numbers for NVIDIA's The post MLPerf Inference v6.1: 5.7x Per-Accelerator Gains, a 512-GPU Run, and Vera Rubin's First Peer-Reviewed Numbers appeared first on StorageReview.com. ## Sparks Fly: NVIDIA Accelerates Local AI at IFA 2026 DevFeed: [Sparks Fly: NVIDIA Accelerates Local AI at IFA 2026]() Original publisher: [Read original article]() Author: Gerardo Delgado Published: 2026-09-03T16:00:59Z Content type: news Language: en Sources: [NVIDIA Blog]() Topics: [Local AI](), [Inference](), [Hardware](), [Inference Performance](), [Mixture of Experts (MoE)]() Tags: [agentic-ai](), [agents](), [ai](), [ai-inference](), [announcements](), [artificial-intelligence](), [dgx-spark](), [hardware](), [inference](), [llama-cpp](), [local-ai](), [mixture-of-experts-moe](), [nvidia](), [nvidia-rtx](), [open-source](), [rtx-ai-garage](), [rtx-spark](), [video](), [vllm]() ### AI overview NVIDIA announces local-AI updates at IFA 2026, including agent tooling, faster local inference, RTX Spark Windows PCs, and locally runnable models for agentic, coding, and video-generation workloads. ### Source excerpt Frontier intelligence is going local. At IFA 2026, NVIDIA, Microsoft and its partners are teaming up to provide faster inference and new tools that make agents easier to set up and run locally on NVIDIA hardware. New compact NVIDIA RTX Spark Windows PCs are also coming in October to give AI enthusiasts, developers and creators [...] ## Fine-tuning a 350M Model for Better Structured Outputs in 100 GRPO Steps DevFeed: [Fine-tuning a 350M Model for Better Structured Outputs in 100 GRPO Steps]() Original publisher: [Read original article]() Author: Leonie Monigatti; ben burtenshaw; Sergio Paniego Published: 2026-09-03T00:00:00Z Content type: tutorial Language: en Sources: [Hugging Face - Blog]() Topics: [AI Chat]() Tags: [artificial-intelligence](), [benchmark](), [community](), [evaluation](), [fine-tuning](), [gpu](), [grpo](), [guide](), [json](), [llama-cpp](), [llm](), [model](), [nlp](), [open-source](), [python](), [rl](), [training](), [trl]() ### AI overview A tutorial on fine-tuning a 350M language model with GRPO to improve structured-output and JSON Schema compliance, then evaluating it on the IFStruct benchmark. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Up to 3.2x Faster Inference with LFM2.5-DSpark DevFeed: [Up to 3.2x Faster Inference with LFM2.5-DSpark]() Original publisher: [Read original article]() Author: Xx; Leonie Monigatti; Fernando Fernandes Neto; Tarek Dakhran; Nathan Ranchin Published: 2026-08-20T16:52:57Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [Inference](), [Large Language Model](), [Latency](), [Streaming](), [sglang](), [GPU](), [Code]() Tags: [agentic](), [benchmark](), [code](), [gpu](), [inference](), [latency](), [llama](), [llama-cpp](), [llm](), [memory](), [on-device](), [streaming]() ### AI overview The article presents LFM2.5-DSpark, a speculative decoding approach that accelerates LLM inference by using a lightweight draft model and verifier. It reports up to 3.18x GPU throughput improvement and up to 2.87x on-device improvement, along with reduced function-calling latency and support for llama.cpp and SGLang. ### Source excerpt - Faster inference: up to 3.18 throughput improvement on a GPU and up to 2.87x on-device. - Toward on-device agentic inference: cuts function-calling latency by 57% on average for LFM2.5-2.6B - Day-one support for llama.cpp and SGLang: LFM-compatible DSpark integration is open-sourced upstream The decode phase in LLM inference is traditionally memory-bound. Most latency comes from streaming weights from DRAM into SRAM, not from intense computation. ## Qwen 3.8 27B is excellent, but it defaults to wildly overthinking things DevFeed: [Qwen 3.8 27B is excellent, but it defaults to wildly overthinking things]() Original publisher: [Read original article]() Author: Simon Willison Published: 2026-08-16T22:00:39Z Content type: opinion Language: en Sources: [Simon Willison]() Topics: [qwen](), [Large Language Model](), [Hardware](), [DGX Spark](), [NVIDIA DGX](), [SVG]() Tags: [ai](), [ai-2-235](), [ai-in-china](), [ai-in-china-108](), [coding-agents](), [coding-agents-248](), [cost](), [dgx-spark](), [generative-ai](), [generative-ai-1-981](), [llama-cpp](), [llama-cpp-29](), [llm](), [llm-reasoning](), [llm-reasoning-103](), [llm-release](), [llm-release-231](), [llms](), [llms-1-947](), [lm-studio](), [lm-studio-23](), [local-llms](), [local-llms-164](), [nvidia-dgx](), [nvidia-spark](), [nvidia-spark-6](), [pelican-riding-a-bicycle](), [pelican-riding-a-bicycle-142](), [pi](), [pi-6](), [qwen](), [qwen-61](), [reasoning](), [speed](), [svg]() ### AI overview Simon Willison evaluates Qwen 3.8 27B, a vision-capable 27-billion-parameter LLM that can run locally on suitable hardware. He finds that its default xhigh reasoning setting consumes substantial context and time, while adjusting the reasoning effort and increasing the context limit improves practicality. He also reports strong results generating an SVG locally. ### Source excerpt Friday's big release was Qwen 3.8 27B, an Apache 2 licensed 27B parameter vision-capable LLM from Alibaba's Qwen research lab. I've been looking forward to this one: 27B is an excellent size for running a model on a reasonably specced laptop, and its predecessor Qwen 3.6 27B was impressive. Qwen's self-reported benchmarks for this model are eye-opening. They show a boost from both Qwen 3.6 27B and the closed-weight Qwen 3.7-Plus, which was one of Qwen's strongest models of any size as recently as May this year. It will be interesting to hear what independent benchmarks have to say about the model. I've been running the model on two different machines: my 128GB M5 Max MacBook Pro, and an NVIDIA DGX Spark. On both machines I'm running LM Studio and their 17GB Q4_K_M quantized build. I also tried using llama-server directly on the Spark. The default of extra high results in spectacular over-thinking Qwen's documentation describes the model as defaulting to xhigh for the reasoning effort, and the LM Studio GGUF I've been trying preserves that default: Qwen3.8 comes with official support for reasoning_effort, which can be used to adjust reasoning depth and control cost: xhigh (default): for complex tasks demanding thorough analysis medium: balancing accuracy and speed low: efficient reasoning optimizing for speed and cost This is a hilarious default. It's absolutely not a good way to run the model, especially on consumer hardware. I've been finding the results extremely entertaining. I quickly ran into problems with LM Studio's default context limit of 8,192 tokens - Qwen was using them all up thinking about even the most mundane of problems. I loaded the model with the full 262,144 maximum context length and that problem went away. Here's the pelican riding a bicycle SVG I got from my first attempt with that increased context length. It took 21 minutes to generate, using 22,276 reasoning tokens to produce 3,223 tokens of output. You can read the reasoning trace here. Th ## Meta is back with Muse Glimmer: local, agentic, multimodal, and open source DevFeed: [Meta is back with Muse Glimmer: local, agentic, multimodal, and open source]() Original publisher: [Read original article]() Author: Pedro Cuenca; merve; ben burtenshaw; Aritra Roy Gosthipaty Published: 2026-08-10T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [vlm](), [multimodal](), [Artificial Intelligence](), [hugging face](), [Benchmark](), [coding]() Tags: [agentic](), [artificial-intelligence](), [benchmark](), [hub](), [hugging-face](), [images](), [llama-cpp](), [llms](), [local](), [meta](), [model](), [multimodal](), [muse](), [open-source](), [videos](), [vllm](), [vlm](), [vlms]() ### AI overview Hugging Face presents Muse Glimmer, a local, agentic, multimodal, open-source 30B-parameter vision-language model developed with Meta. The article outlines its vision and language architecture, benchmark context, optional speculative decoding for faster generation, and support for both images and videos. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Welcome Inkling by Thinking Machines DevFeed: [Welcome Inkling by Thinking Machines]() Original publisher: [Read original article]() Author: ben burtenshaw; merve; Pedro Cuenca; Aritra Roy Gosthipaty; Andres Marafioti Published: 2026-07-15T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [Large Language Model](), [Mixture of Experts (MoE)](), [multimodal](), [hugging face](), [Inference](), [Deployment](), [Fine-tuning](), [Open Source](), [sglang]() Tags: [announcement](), [artificial-intelligence](), [audio](), [context-window](), [deployment](), [fine-tuning](), [generation](), [inference](), [inference-endpoints](), [llama-cpp](), [llms](), [mixture-of-experts](), [mixture-of-experts-moe](), [moe](), [multimodal](), [open-source](), [sglang](), [transformers](), [vllm]() ### AI overview Thinking Machines Lab's Inkling is presented as a large open multimodal language model that accepts image, text, and audio inputs. The article covers its mixture-of-experts architecture, million-token context window, reasoning across modalities, fine-tuning use cases, model variants, and deployment through Hugging Face Inference Endpoints and inference frameworks. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Native-speed vLLM transformers modeling backend DevFeed: [Native-speed vLLM transformers modeling backend]() Original publisher: [Read original article]() Author: Harry Mellor; Lysandre Published: 2026-07-08T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [vllm](), [Inference Performance](), [Inference](), [LLMs](), [Machine learning](), [Open Source]() Tags: [architecture](), [batching](), [gpu](), [inference](), [inference-performance](), [llama](), [llama-cpp](), [llms](), [mixture-of-experts](), [mlx](), [open-source](), [optimization](), [sglang](), [transformers](), [vllm]() ### AI overview This article describes the transformers modeling backend for vLLM and its role in serving Hugging Face models without requiring model authors to port their implementations. It explains how transformers supplies modeling code while vLLM contributes optimized inference techniques, and outlines comparisons with vLLM's native implementations across several Qwen3 model configurations. It also discusses inference-performance factors including GPU parallelization, compilation, fused kernels, and custom attention kernels. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Sampling args in llama-server DevFeed: [Sampling args in llama-server]() Original publisher: [Read original article]() Author: Alex Ewerlöf Published: 2026-07-01T18:28:35Z Content type: tutorial Language: en Sources: [Alex Ewerlof Notes]() Topics: [llama.cpp](), [Large Language Model](), [Inference](), [quantization](), [Benchmark]() Tags: [benchmark](), [hallucinations](), [inference](), [llama-cpp](), [llm](), [quantization]() ### AI overview A reference guide to llama.cpp sampling and repetition parameters, covering common failure modes in local and quantized language models, parameter ranges and defaults, workload-specific tuning, modern sampling methods, and ways to accelerate experimentation. ### Source excerpt Reducing repetition, hallucinations, degradation, while making inference faster! ## Using local LLMs for agentic coding DevFeed: [Using local LLMs for agentic coding]() Original publisher: [Read original article]() Author: Alex Ewerlöf Published: 2026-06-04T09:01:34Z Content type: tutorial Language: en Sources: [Alex Ewerlof Notes]() Topics: [coding](), [agentic-coding](), [Large Language Model](), [GitHub Copilot CLI](), [Linux](), [llama.cpp](), [Ollama](), [NVIDIA RTX]() Tags: [agentic-coding](), [agents](), [ai](), [claude](), [coding](), [github](), [linux](), [llama](), [llama-cpp](), [llms](), [local](), [local-llms](), [microsoft](), [nvidia](), [nvidia-rtx](), [ollama](), [quality](), [rocm](), [state]() ### AI overview A practical guide to using local language models for agentic coding. It covers running local models, configuring coding agents such as Copilot and Pi, and evaluating local models against cloud models in terms of cost, privacy, tooling, and performance. ### Source excerpt AI honeymoon pricing is over, but your work is not ## Use Your Mac for AI Agents: Self-Host Gemma 4 12 B with Pulumi and Tailscale DevFeed: [Use Your Mac for AI Agents: Self-Host Gemma 4 12 B with Pulumi and Tailscale]() Original publisher: [Read original article]() Author: Pablo Seibelt Published: 2026-06-04T00:00:00Z Content type: tutorial Language: en Sources: [Pulumi]() Topics: [gemma4](), [llama.cpp](), [Local AI](), [Infrastructure as code](), [macOS](), [On-device AI](), [multimodal]() Tags: [ai](), [download](), [gemma-4](), [gemma4](), [gpu](), [infrastructure-as-code](), [kubernetes](), [llama-cpp](), [macos](), [multimodal](), [python](), [tailscale](), [tutorials]() ### AI overview A tutorial for self-hosting Gemma 4 12 B on a modern Mac using llama.cpp with Apple Metal acceleration. It combines host-native inference with a local Kubernetes cluster, Pulumi infrastructure as code, and Tailscale for secure access, and reports validation results on a MacBook Pro with an Apple M3 Max and 36 GB RAM. ### Source excerpt If you run AI tools and agents, you've probably accepted three tradeoffs: your data leaves your network, you can't work offline, and your bill scales with usage. Open-weight models now run well on consumer hardware. Once the model is on your machine, your data stays local, inference works offline, and tokens cost nothing. If you own a modern Mac, you can run a high-quality model yourself. Gemma 4 is an open-weights model family from Google. This post focuses on Gemma 4 12 B, released in June 2026, using Unsloth's Q8_0 GGUF. The 12 B model fits comfortably on a modern Mac while leaving enough headroom for local llama.cpp and a chat UI. We'll use llama.cpp for host-native inference, k3d for a local Kubernetes cluster, Pulumi for infrastructure as code, and Tailscale for secure access. Prerequisites This setup was validated on the following hardware: macOS 26 Tahoe, version 26.5 MacBook Pro with Apple M3 Max 36 GB RAM On this machine, llama.cpp reported about 20 output tokens per second for a 160-token validation response with unsloth/gemma-4-12b-it-GGUF, gemma-4-12b-it-Q8_0.gguf, and a 131,072-token context. Sustained throughput varies by prompt length, thermal state, and llama.cpp settings. You'll need brew, docker, pulumi, and tailscale installed. We'll also install k3d during the process. Run Gemma 4 with host-native llama.cpp We use llama.cpp directly on macOS to leverage Apple Metal acceleration. Running the LLM on the host is more efficient than trying to pass GPU access into a local Kubernetes VM. Install the build tools: brew install cmake git Then build llama.cpp from source and download the multimodal projector. In validation, Homebrew llama.cpp 9430 could run text inference, but it could not load the new Gemma 4 12 B projector and failed with unknown projector type: gemma4uv. Building current llama.cpp from source fixed that. llm_home="$HOME/pulumi-gemma4-llm" mkdir -p "$llm_home/models" "$llm_home/logs" if [ ! -d "$llm_home/llama.cpp/.git" ]; then git clon ## WMMA guide for AMD RDNA 4 architecture GPUs - part 3 DevFeed: [WMMA guide for AMD RDNA 4 architecture GPUs - part 3]() Original publisher: [Read original article]() Author: Hui Zhang Published: 2026-06-02T19:02:00Z Content type: tutorial Language: en Sources: [AMD GPUOpen]() Topics: [Code](), [Optimization](), [llama.cpp](), [Warp]() Tags: [3](), [amd](), [architecture](), [code](), [cpp](), [cuda](), [getting-started](), [guide](), [hip](), [implement](), [llama-cpp](), [maths](), [memory](), [ml](), [optimization](), [performance](), [rdna](), [reference-guide](), [technical-article](), [technical-articles](), [user-guides-manuals]() ### AI overview This tutorial explains how to implement an in-register matrix transpose using the WMMA layout on AMD RDNA 4 GPUs. It describes an identity-matrix approach that avoids additional memory operations and notes its use in Llama.cpp for Flash Attention. ### Source excerpt Learn how to implement fast in-register matrix transpose on AMD RDNA™ 4 architecture GPUs with a WMMA-based identity trick, delivering a lightweight, memory-free alternative proven in Llama.cpp. ## AI, OAuth, And Other Platform APIs In The Core DevFeed: [AI, OAuth, And Other Platform APIs In The Core]() Original publisher: [Read original article]() Author: Shai Almog Published: 2026-05-31T00:00:00Z Content type: article Language: en Sources: [CodeName One]() Topics: [Artificial Intelligence](), [OAuth](), [OpenID connect (OIDC)](), [WebAuthn](), [Passkeys](), [Framework](), [API](), [Large Language Model](), [Ollama](), [llama.cpp](), [vllm](), [SQLite]() Tags: [agent-skill](), [ai](), [api](), [authentication](), [function-calling](), [llama-cpp](), [llm](), [oauth](), [oidc](), [ollama](), [sqlite](), [streaming](), [vllm]() ### AI overview This follow-up release article describes platform APIs moved into the framework core, including a first-class LLM client and ChatView, OAuth and OIDC authentication, WebAuthn passkeys, WiFi and connectivity APIs, and share-sheet callbacks. It also outlines streaming chat, tool calls, embeddings, image generation, and local-model support through Ollama-compatible endpoints. ### Source excerpt Deeper AI integration in the framework core, modern authentication via OAuth / OIDC and WebAuthn passkeys driven from the system browser, and a few smaller additions alongside. ## How I Run My Daily Standup on a Local LLM: Git, Gemma 4, llama.cpp and Telegram DevFeed: [How I Run My Daily Standup on a Local LLM: Git, Gemma 4, llama.cpp and Telegram]() Original publisher: [Read original article]() Author: Ivan Morgillo Published: 2026-05-27T09:51:06Z Content type: tutorial Language: en Sources: [Ivan Morgillo]() Topics: [gemma4](), [llama.cpp](), [Git](), [Large Language Model](), [API](), [Claude]() Tags: [anthropic](), [api](), [claude](), [gemma](), [gemma-4](), [git](), [llama-cpp](), [llm](), [local](), [local-llm](), [sdk](), [telegram]() ### AI overview A developer describes replacing a cloud-based daily standup pipeline with a local Gemma 4 E4B model running on llama.cpp. The pipeline reads Git commits, formats them, and posts the summary to Telegram. The article also explains how empty API-key settings caused several automations to use personal Claude subscription authentication instead of the Anthropic API. ### Source excerpt A real-world pipeline that scans my Git repos every morning, formats the day's commits with a local Gemma 4 E4B on llama.cpp, and posts the summary to Telegram -- no cloud, no API key, no AUP warnings. ## Reachy Mini goes fully local DevFeed: [Reachy Mini goes fully local]() Original publisher: [Read original article]() Author: Amir Mahla; Andres Marafioti Published: 2026-05-27T00:00:00Z Content type: tutorial Language: en Sources: [Hugging Face - Blog]() Topics: [reachy](), [gemma4](), [llama.cpp](), [Large Language Model](), [speech-to-speech](), [hugging face](), [WebSocket]() Tags: [api](), [artificial-intelligence](), [audio](), [blog](), [gemma](), [guide](), [how-to](), [llama](), [llama-cpp](), [llm](), [local](), [local-ai](), [open-source](), [reachy](), [robotics](), [server](), [speech](), [speech-to-speech]() ### AI overview A tutorial for running fully local conversations with a Reachy Mini robot. It describes a cascaded VAD, speech-to-text, LLM, and text-to-speech pipeline using llama.cpp with Gemma 4, Silero VAD, Parakeet-TDT STT, and Qwen3-TTS, connected through a Realtime API-compatible WebSocket. The setup avoids cloud services, API keys, and sending data off the local machine. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Why and How to Run Local Models in Zed DevFeed: [Why and How to Run Local Models in Zed]() Original publisher: [Read original article]() Author: Cameron Mcloughlin Published: 2026-05-19T00:00:00Z Content type: tutorial Language: en Sources: [Zed Industries - Blog]() Topics: [Local AI](), [LLMs](), [llama.cpp](), [Ollama](), [data]() Tags: [developer](), [hardware](), [llama](), [llama-cpp](), [llms](), [local](), [local-ai](), [ollama]() ### AI overview This article explains why developers may choose local models in Zed, focusing on privacy, cost, control, and availability. It also discusses their limitations compared with cloud-hosted frontier models and introduces setup guidance. ### Source excerpt You can run local AI models in Zed to get better performance and control over your data. Here's how. ## Welcome Gemma 4: Frontier multimodal intelligence on device DevFeed: [Welcome Gemma 4: Frontier multimodal intelligence on device]() Original publisher: [Read original article]() Author: merve; Pedro Cuenca; Sergio Paniego; ben burtenshaw; Steven Zheng; Alvaro Bartolome; Nathan Habib Published: 2026-04-02T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [gemma4](), [multimodal](), [Open Source](), [long-context](), [quantization](), [llama.cpp](), [MLX](), [Rust](), [webgpu]() Tags: [audio](), [benchmarks](), [fine-tuning](), [gemma](), [gemma4](), [generation](), [google](), [guide](), [llama](), [llama-cpp](), [local](), [long-context](), [mlx](), [models](), [moe](), [multimodal](), [on-device](), [open](), [open-source](), [quantization](), [science](), [speed]() ### AI overview Gemma 4 is presented as an open, Apache 2-licensed family of multimodal models supporting image, video, text, and, for selected variants, audio inputs. The article highlights on-device deployment, long-context support, compatibility across libraries and devices, quantization, benchmark results, and implementation options including llama.cpp, MLX, WebGPU, and Rust. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Liberate your OpenClaw DevFeed: [Liberate your OpenClaw]() Original publisher: [Read original article]() Author: Clem 🤗; ben burtenshaw; Pedro Cuenca; Jeff Boudier; merve; Niels Rogge; Victor Mustar; Mishig ᠮᠢᠰᠾᠢᠭ Published: 2026-03-27T00:00:00Z Content type: tutorial Language: en Sources: [Hugging Face - Blog]() Topics: [OpenClaw](), [Open Source](), [hugging face](), [inference-providers](), [llama.cpp]() Tags: [agents](), [guide](), [hugging-face](), [inference-providers](), [llama-cpp](), [local-ai](), [open-source](), [openclaw](), [privacy](), [server]() ### AI overview This tutorial explains how to restore OpenClaw agents using open models through Hugging Face Inference Providers or by running models locally with llama.cpp. It compares hosted and local approaches, covering privacy, cost, hardware, model selection, configuration, and local server setup. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Local LLM Inference : llama.cpp, GGUF, Quantizations and GGML Explained DevFeed: [Local LLM Inference : llama.cpp, GGUF, Quantizations and GGML Explained]() Original publisher: [Read original article]() Author: Alex Razvant Published: 2026-03-03T11:31:04Z Content type: tutorial Language: en Sources: [Neural Bits]() Topics: [llama.cpp](), [ggml](), [Inference](), [Large Language Model](), [Local AI](), [quantization]() Tags: [backend](), [cross-platform](), [efficiently](), [embedded](), [format](), [ggml](), [inference](), [llama-cpp](), [llm](), [local-llm](), [model]() ### AI overview A practical guide to local LLM inference with llama.cpp, explaining how the GGUF model format, GGML backend concepts, quantization, and inference workflows fit together for efficient execution on edge devices. ### Source excerpt Learn how the llama.cpp runtime, GGML backend concepts, and GGUF model format fit together for fast local inference across devices. ## GTX 1080 Ti for Local LLM DevFeed: [GTX 1080 Ti for Local LLM]() Original publisher: [Read original article]() Published: 2026-02-28T23:33:54Z Content type: article Language: en Sources: [Ariya Hidayat]() Topics: [GPU](), [LLMs](), [llama.cpp](), [Nvidia](), [CUDA](), [Inference Performance](), [Retrieval Augmented Generation (RAG)]() Tags: [cmake](), [command-line](), [cuda](), [gcc](), [gemma](), [gpu](), [inference](), [inference-performance](), [llama](), [llama-cpp](), [llms](), [nvidia]() ### AI overview This article evaluates the eight-year-old NVIDIA GTX 1080 Ti as an affordable option for running local LLMs. It explains how its 11 GB of VRAM can support quantized models and relevant context windows, then describes measuring prompt-processing and token-generation performance with CUDA-enabled llama.cpp. ### Source excerpt Despite being over eight years old, the NVIDIA GTX 1080 Ti remains a compelling choice for enthusiasts keen on running LLM locally. ## GGML and llama.cpp join HF to ensure the long-term progress of Local AI DevFeed: [GGML and llama.cpp join HF to ensure the long-term progress of Local AI]() Original publisher: [Read original article]() Author: Georgi Gerganov; Xuan-Son Nguyen; Aleksander Grygier; Lysandre; Victor Mustar; Julien Chaumond Published: 2026-02-20T00:00:00Z Content type: news Language: en Sources: [Hugging Face - Blog]() Topics: [llama.cpp](), [Local AI](), [Inference](), [Open Source](), [model-deployment](), [Artificial Intelligence](), [User Experience]() Tags: [community](), [devices](), [ggml](), [inference](), [llama-cpp](), [llm](), [local-ai](), [open-source](), [superintelligence](), [user-experience]() ### AI overview Georgi Gerganov and the ggml team are joining Hugging Face to support the llama.cpp and ggml communities while retaining autonomy over technical direction and community leadership. The collaboration will provide long-term resources, improve integration between model definitions and llama.cpp, simplify packaging and user experience, and expand access to efficient local inference on devices. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## New in llama.cpp: Model Management DevFeed: [New in llama.cpp: Model Management]() Original publisher: [Read original article]() Author: Xuan-Son Nguyen; Victor Mustar Published: 2025-12-11T15:47:44Z Content type: release Language: en Sources: [Hugging Face - Blog]() Topics: [AI, ML & Data Engineering](), [Processes]() Tags: [cache](), [gpu](), [llama-cpp](), [llms](), [management](), [models](), [process](), [routing](), [server]() ### AI overview llama.cpp introduces model management for its local, OpenAI-compatible LLM server. Router mode discovers GGUF models, loads them on demand, evicts least-recently-used models, and routes requests by model field. ### Source excerpt Reminder: llama.cpp server is a lightweight, OpenAI-compatible HTTP server for running LLMs locally. This feature was a popular request to bring Ollama-style model management to llama.cpp. It uses a multi-process architecture where each model runs in its own process, so if one model crashes, others remain unaffected. Start the server in router mode by not specifying a model: This auto-discovers models from your llama.cpp cache ( or ). ## Introducing AnyLanguageModel: One API for Local and Remote LLMs on Apple Platforms DevFeed: [Introducing AnyLanguageModel: One API for Local and Remote LLMs on Apple Platforms]() Original publisher: [Read original article]() Author: Mattt Published: 2025-11-20T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [SDKs](), [AI Chat](), [Google AI]() Tags: [anthropic](), [api](), [apple](), [community](), [coreml](), [gemini](), [llama-cpp](), [llm](), [llms](), [local](), [mlx](), [open-source](), [openai](), [remote](), [swift]() ### AI overview AnyLanguageModel is a Swift package that lets Apple-platform developers use a common Foundation Models-style API across local and remote LLM providers. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Granite 4.0 Nano: Just how small can you go? DevFeed: [Granite 4.0 Nano: Just how small can you go?]() Original publisher: [Read original article]() Author: Kate Soule; Rameswar Panda Published: 2025-10-28T14:59:38Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [ibm](), [Large Language Model](), [Model Development](), [Open Source Models & Datasets](), [llama.cpp](), [MLX](), [vllm](), [Transformer](), [Loop Engineering](), [gemma](), [qwen](), [Google]() Tags: [apache](), [architecture](), [benchmarks](), [code](), [data](), [development](), [gemma](), [google](), [ibm](), [language-models](), [llama](), [llama-cpp](), [math](), [mlx](), [model](), [model-development](), [models](), [on-device](), [qwen](), [training]() ### AI overview IBM introduces Granite 4.0 Nano, a family of compact language models for edge and on-device applications. The release includes hybrid-SSM and traditional transformer variants ranging from roughly 350M to 1.5B parameters, supports vLLM, llama.cpp, and MLX, and is released under the Apache 2.0 license. The article reports strong performance across knowledge, math, code, safety, instruction-following, and tool-calling benchmarks. ### Source excerpt Today we are excited to share Granite 4.0 Nano, our smallest models yet, released as part of IBM's Granite 4.0 model family. Designed for the edge and on-device applications, these models demonstrate excellent performance for their size and represent IBM's continued commitment to develop powerful, useful, models that don't require hundreds of billions of parameters to get the job done. [Next page]()