# onnx Published articles for onnx. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## A Practical Loop for Optimizing GPU Inference Throughput DevFeed: [A Practical Loop for Optimizing GPU Inference Throughput]() Original publisher: [Read original article]() Author: Andrei Potapkin Published: 2026-05-01T17:57:59Z Content type: article Language: en Sources: [Bumble Tech]() Topics: [Inference](), [Optimization](), [GPU](), [Benchmark](), [TensorRT](), [onnx](), [Cloud](), [migration]() Tags: [benchmark](), [cloud](), [cloud-migration](), [gpu](), [inference](), [nvidia-nsight-systems](), [onnx](), [optimization](), [python](), [tensorrt]() ### AI overview This practical article describes a production-focused loop for optimizing GPU inference: benchmark under representative load, identify the current throughput bottleneck, fix it, and verify the operational outcome. In the described ML moderation service, TensorRT, BF16, and an attention-subgraph rewrite increased throughput from roughly 150 to roughly 1100 requests per second in the same saturated benchmark. ### Source excerpt A practical loop for optimizing GPU inference without guessing It wasn't an incident. That made it interesting. A few days after a cloud migration, one of our highest-traffic ML services looked healthy. Traffic was flowing. Latency was fine. The rollout had gone the way you want these things to go: boringly. And yet a Slack thread started anyway. Not because anything was broken. Because something had become easier to see. On-prem, GPU efficiency tends to get folded into bigger conversations: capacity planning, hardware timelines, headroom for the next model. In the cloud, the same question sharpens. Throughput, utilization, and cost sit much closer together. A service can be perfectly healthy and still look expensive. That was our situation. We had a frontend text moderation model based on mT5, served through NVIDIA Triton as a two-component ensemble: a Python tokenizer and a static ONNX model. The service was stable, but throughput per GPU was much lower than our back-of-the-envelope estimates suggested Moments like that attract the usual optimization advice: Batch more aggressively. Tune Triton. Rewrite the Python stages. Enable TensorRT. Drop precision. None of that advice is silly. Some of it is very good. The problem is that it usually arrives before the diagnosis. GPU systems are awkward that way. By the time you go hunting for "the optimization", you are often already asking the wrong question. There usually isn't one magical fix. There is a bottleneck somewhere in the stack, and then another one behind it, and usually another one behind that. The loop that kept us honest was simple: TL;DR Benchmark the system under production-shaped load. Find the bottleneck limiting throughput right now. Fix that bottleneck. Check whether the operational outcome changed. In our case, that loop took us from roughly 150 requests per second to roughly 1100 in the same saturated benchmark: a bit over 7x baseline. TensorRT delivered the first big jump. BF16 moved the ceiling aga ## Scaling a Vespa Application: Feeding Fast and Furiously DevFeed: [Scaling a Vespa Application: Feeding Fast and Furiously]() Original publisher: [Read original article]() Author: Kai Borgen Published: 2026-04-28T00:00:00Z Content type: tutorial Language: en Sources: [Vespa Blog]() Topics: [AI search](), [Deployment](), [datasets](), [information retrieval](), [Cloud](), [Homebrew](), [Command-line interface](), [onnx](), [optimum](), [XML]() Tags: [ai](), [ai-search](), [cli](), [cloud](), [dataset](), [genai](), [how-to](), [information-retrieval](), [install](), [metrics](), [onnx](), [optimum](), [performance](), [rag](), [ranking](), [tensors](), [tutorial]() ### AI overview This tutorial demonstrates how to scale a Vespa application while feeding the full MS_marco passages dataset. It covers preparing the dataset, configuring access, deploying a sample application, and using scaling and metrics to improve feed throughput and performance. ### Source excerpt A tutorial on how to scale the resources in a Vespa application to increase feed throughput. Using the metrics dashboard for informed and optimised scaling. ## Using Large ONNX Models with External Data in Vespa Embedders DevFeed: [Using Large ONNX Models with External Data in Vespa Embedders]() Original publisher: [Read original article]() Author: bjorncs thomas Published: 2026-03-27T00:00:00Z Content type: tutorial Language: en Sources: [Vespa Blog]() Topics: [onnx](), [Cloud]() Tags: [authentication](), [cloud](), [embedding](), [files](), [how-to](), [models](), [onnx](), [protocol](), [serialization-format]() ### AI overview Vespa embedders now support large ONNX models whose weights are stored in external data files. Starting with Vespa 8.544, Vespa automatically downloads referenced external files when loading URL-based models, with support for private models through propagated authentication tokens. The feature is limited to embedders and supported model references. ### Source excerpt Many ONNX models exceed the 2GB protobuf limit and store weights in external data files. Vespa now supports these models for embedders. ## Gemma 3 AI model in Clojure DevFeed: [Gemma 3 AI model in Clojure]() Original publisher: [Read original article]() Published: 2025-12-09T22:35:00Z Content type: tutorial Language: en Sources: [Dragan Djuric]() Topics: [Clojure](), [Artificial Intelligence](), [Inference](), [Large Language Model](), [onnx](), [gemma]() Tags: [3](), [ai](), [clojure](), [code](), [deep](), [diamond](), [gemma](), [inference](), [llms](), [onnx]() ### AI overview This tutorial demonstrates loading and running a one-step Gemma 3 inference in Clojure through the ONNX runtime integration in Deep Diamond. It configures a smaller one-billion-parameter model, uses main-memory tensors with the oneDNN engine, and explains that the demonstrated output is a next-token tensor rather than a complete generated response. ### Source excerpt Recently I've been working on the ONNX runtime integration into Deep Diamond, backed by the grant sponsored by the Clojurists Together Foundation. In the past few articles, we've seen how ONNX models are integrated into Deep Diamond, using only a single function onnx, with almost no need for additional configuration (which is available). I used a simple MNIST model in the demonstration. But, can we now load and run the inference on the real deal models, such as the open LLMs from the Hugging Face, for example? Let's see! The Hugging Face model card has this to say about Gemma 3: "Gemma is a family of lightweight, state-of-the-art open models from Google, built from the same research and technology used to create the Gemini models." (etc., etc.) So, it seems to be something worth trying. I'll try to be brief, and skip the unnecessary talk. Let's just show the code, which I've just lifted up and adapted from the Diamond's midje tests. What we need for this? First, decide on the backend engine; this time we'll use tensors in main memory backed up by the oneDNN engine (DNNL). (def fact (dnnl-factory)) (def neand-fact (neanderthal-factory fact)) Next, load and configure a particular flavor of Gemma 3 (a smaller one, only 1 billion parameters). The onnx function creates a generalized blueprint, which can create the actual functions when evaluated with the specific input tensors. (def onnx-bp (onnx fact "data/gemma-3-1b-it-ONNX-GQA/onnx/model.onnx" {:options (-> (options) (override-dimension! "batch_size" 1) (override-dimension! "sequence_length" 1) (override-dimension! "past_sequence_length" 1) (override-dimension! "total_sequence_length" 1))}) Gemma 3 has 63 inputs and 61 outputs. We'll need to provide these, but even here we can automate some parts with Clojure, since past-key values are pretty uniform. We only need to provide inputs, while the engine can create the outputs for us. (def src-tz (tensor fact [1 1 28 28] :float :nchw)) (def input-ids (tensor neand-fact [1 ## Four ways to run ONNX models on a GPU with CUDA in Clojure DevFeed: [Four ways to run ONNX models on a GPU with CUDA in Clojure]() Original publisher: [Read original article]() Published: 2025-11-09T17:49:00Z Content type: tutorial Language: en Sources: [Dragan Djuric]() Topics: [CUDA](), [GPU](), [onnx](), [Clojure](), [Artificial Intelligence](), [Deep neural networks]() Tags: [ai](), [clojure](), [code](), [cuda](), [deep](), [diamond](), [examples](), [gpu](), [image-recognition](), [model](), [neural-networks](), [onnx](), [tensors]() ### AI overview A tutorial presents four ways to run ONNX models on a GPU with CUDA using Clojure libraries including Diamond ONNX RT, Deep Diamond, and ClojureCUDA. It covers GPU tensor backends and a configuration that keeps input and output tensors in main memory while executing the model on the GPU. ### Source excerpt Two weeks ago, I announced a new Clojure ML library, Diamond ONNX RT, which integrates ONNX Runtime into Deep Diamond. In that post, we explored the classic Hello World example of Neural Networks, MNIST handwritten image recognition, step-by-step. We run that example on the CPU, from main memory. The next logical step is to execute this stuff on the GPU. You'll see that with a little help of ClojureCUDA and Deep Diamond built-in CUDA machinery, this is both easy and simple, requiring almost no effort from a curious Clojure programmer. But don't just trust me; let's fire up your REPL, and we can continue together. Here's how you can evaluate this directly in your REPL (you can use the Hello World that is provided in the ./examples sub-folder of Diamond ONNX RT as a springboard). Require Diamond's namespaces First things first, we refer functions that we're going to use. (require '[uncomplicate.commons.core :refer [with-release]] '[uncomplicate.neanderthal.core :refer [transfer! iamax native]] '[uncomplicate.diamond [tensor :refer [tensor with-diamond]] [dnn :refer [network]] [onnxrt :refer [onnx]]] '[uncomplicate.diamond.internal.dnnl.factory :refer [dnnl-factory]] '[uncomplicate.diamond.internal.cudnn.factory :refer [cudnn-factory]] '[hello-world.native :refer [input-desc input-tz mnist-onnx]]) None of the following ways to run CUDA models has preference, you use the one that best suits your needs. Way one One of the ways to run ONNX models on your GPU is to simply use Deep Diamond's cuDNN factory as the backend for your tensors. Then, the machinery recognizes what you need and proceeds doing everything on the GPU, using the right stream for tensors, Deep Diamond operations, and ONNX Runtime operations. This looks exactly the same as any other Deep Diamond example from this blog or the DLFP book. (with-diamond cudnn-factory [] (with-release [cuda-input-tz (tensor input-desc) mnist (network cuda-input-tz [mnist-onnx]) classify! (mnist cuda-input-tz)] (transfer! input ## Clojure API for Running ONNX Models with ONNX Runtime DevFeed: [Clojure API for Running ONNX Models with ONNX Runtime]() Original publisher: [Read original article]() Published: 2025-10-26T15:56:00Z Content type: tutorial Language: en Sources: [Dragan Djuric]() Topics: [Clojure](), [onnx](), [AI Models]() Tags: [ai](), [clojure](), [deep](), [diamond](), [onnx](), [tensors]() ### AI overview The author describes work on a Clojure API for using pre-trained models exported in ONNX format through ONNX Runtime. The approach uses ONNX Runtime's underlying C library rather than Python interoperability. ### Source excerpt Hello, Clojurians! I haven't written here in a long time. Was I tired? Is anybody reading blogs anymore? Who knows. But that was not the main reason. I've been working on several Clojure projects sponsored by the Clojurists Together Foundation. I did a ton of things, but after all this programming, I was kinda tired, and kept slugging when it comes to telling people about the work done! That's not very smart, but you know how it goes... :) But, then, if we don't tell people about awesome software that we have, nobody is going to use it, so finally I had to stop kicking this down the road, sit, and write the first post. It's been long overdue, so expect more posts soon! ONNX Runtime in one line of Clojure The most recent thing I'm currently working on started its life as Clojure ML (again, superthanks to Clojurists Together for sponsoring this). I proposed to create a human-friendly Clojure API for AI/DL/ML models, and back it by the first implementation, in this case based on ONNX Runtime. Of course, it should all be integrated into existing Clojure libraries, and follow the Clojure way of doing stuff as much as possible! The idea is to get an existing, pre-trained ML model previously exported to the ONNX format from whatever technology the authors chose (which in today's world is typically Python and PyTorch), and put it into production in Clojure and JVM. It should be seamless and in-process, without any clunky interoperability, copy, translation, etc. Of course, our Clojure numerical libraries fully support GPU computing, so it goes without saying that we want that, too! Just to be clear, we do not use nor need any Python or Python interop for this, we use the ONNX Runtime's underlying C library. Nice idea, but what parts of this well intended story can we evaluate in our REPLs right now? At least some promising demo? Are we on the trail? To access that AI goodness, we surely have to do a sophisticated dance? Are the steps hard to learn? Do we need to watch careful ## Welcome EmbeddingGemma, Google's new efficient embedding model DevFeed: [Welcome EmbeddingGemma, Google's new efficient embedding model]() Original publisher: [Read original article]() Author: Tom Aarsen; Joshua; Alvaro Bartolome; Aritra Roy Gosthipaty; Pedro Cuenca; Sergio Paniego Published: 2025-09-04T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [Embeddings](), [sentence-transformers](), [Artificial Intelligence](), [Google](), [Retrieval-Augmented Generation](), [Open Source](), [transformers.js](), [Inference](), [onnx](), [dataset](), [LangChain](), [llamaindex]() Tags: [artificial-intelligence](), [benchmark](), [community](), [embedding](), [google](), [guide](), [langchain](), [llamaindex](), [nlp](), [on-device](), [onnx](), [open-source](), [rag](), [sentence-transformers](), [transformers-js]() ### AI overview Google introduces EmbeddingGemma, a compact multilingual embedding model designed for fast, efficient on-device use. The article covers its architecture, training, multilingual capabilities, benchmark performance, framework integrations, and domain fine-tuning for retrieval applications. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Arm & ExecuTorch 0.7: Bringing Generative AI to the masses DevFeed: [Arm & ExecuTorch 0.7: Bringing Generative AI to the masses]() Original publisher: [Read original article]() Author: EricSondhi; Gian Marco Iodice Published: 2025-08-13T14:55:10Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [Arm](), [Generative AI](), [Artificial Intelligence](), [Large Language Model](), [AI Inference](), [cpu](), [Mobile](), [Raspberry Pi](), [Latency]() Tags: [ai](), [ai-accelerator](), [android](), [arm](), [cpu](), [cross-platform](), [developers](), [edge-ai](), [generative](), [generative-ai](), [large-language-models-llms](), [latency](), [llama-cpp](), [mediapipe](), [memory](), [onnx](), [performance](), [raspberry-pi](), [smartphones]() ### AI overview Arm's KleidiAI will be enabled by default in the upcoming ExecuTorch 0.7 beta, providing automatic AI acceleration through ExecuTorch and XNNPack on current and older Arm-based devices. The article explains how this can improve model startup time, latency, and memory efficiency, making Generative AI and large language models more practical on mobile devices and Raspberry Pi 5. It also describes how the Arm SDOT instruction accelerates low-precision matrix multiplication used by LLMs. ### Source excerpt With Arm's recent SME2 announcement, the role of Arm KleidiAI is increasingly clear as Arm's AI accelerator layer powering the next wave of AI. By embedding into widely-used Edge AI frameworks like XNNPack, MediaPipe, MNN, ONNX Runtime, and even llama.cpp, KleidiAI has delivered substantial performance improvements with no code changes required by developers. ## Transformers.js v3: WebGPU Support, New Models & Tasks, and More... DevFeed: [Transformers.js v3: WebGPU Support, New Models & Tasks, and More...]() Original publisher: [Read original article]() Author: Joshua Published: 2024-10-22T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [transformers.js](), [webgpu](), [AI Inference](), [quantization](), [Whisper](), [asr]() Tags: [announcement](), [artificial-intelligence](), [javascript](), [npm](), [onnx](), [open-source](), [quantization](), [transformers](), [transformers-js](), [webgl](), [webgpu]() ### AI overview Transformers.js v3 adds WebGPU acceleration for browser-based machine learning, support for more quantization options and per-module data types, and 120 supported model architectures. The release includes examples for text embeddings, speech recognition with Whisper, image classification, and other models and tasks. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## XLSCOUT Unveils ParaEmbed 2.0: a Powerful Embedding Model Tailored for Patents and IP with Expert Support from Hugging Face DevFeed: [XLSCOUT Unveils ParaEmbed 2.0: a Powerful Embedding Model Tailored for Patents and IP with Expert Support from Hugging Face]() Original publisher: [Read original article]() Author: Andrew Reed; Khushwant Rai Published: 2024-06-25T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [Artificial Intelligence](), [Embeddings](), [Fine-tuning](), [hugging face](), [Open Source](), [Open Source Models & Datasets](), [Inference](), [Google Cloud Platform (GCP)]() Tags: [artificial-intelligence](), [case-studies](), [embedding](), [fine-tuning](), [gcp](), [hugging-face](), [inference](), [llama](), [mixtral](), [onnx](), [open-source]() ### AI overview XLSCOUT describes ParaEmbed 2.0, a proprietary embedding model developed for patent and intellectual-property analysis through collaboration with Hugging Face's Expert Support Program. The model was fine-tuned on expert-curated, multi-domain patent data and reportedly improved accuracy by 23% over ParaEmbed 1.0. The article also describes the use of open-source models, ONNX optimization, and a TorchServe inference server on Google Cloud Platform. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## AI Inference now available in Supabase Edge Functions DevFeed: [AI Inference now available in Supabase Edge Functions]() Original publisher: [Read original article]() Author: Lakshan Perera Published: 2024-04-16T07:00:00Z Content type: article Language: en Sources: [Supabase Blog]() Topics: [Supabase](), [AI Inference](), [Embeddings](), [Large Language Model](), [Ollama](), [transformers.js](), [wasm](), [Edge](), [Rust]() Tags: [ai](), [ai-inference](), [ai-models](), [cross-platform](), [embeddings](), [large-language-models](), [ollama](), [onnx](), [postgres](), [transformers-js](), [wasm]() ### AI overview Supabase Edge Functions now provide a built-in API for running AI inference workloads. The article explains embedding generation with pgvector, planned Large Language Model support through Ollama, and architectural improvements using ONNX Runtime through a native Rust extension to reduce cold-start overhead. ### Source excerpt Use embeddings and large language models on the edge with Supabase Edge Functions. ## KotlinDL 0.3: поддержка ONNX, Object Detection API, 20+ новых моделей в ModelHub, и много новых слоев DevFeed: [KotlinDL 0.3: поддержка ONNX, Object Detection API, 20+ новых моделей в ModelHub, и много новых слоев]() Original publisher: [Read original article]() Author: zaleslaw (JetBrains) Published: 2021-10-04T13:56:53Z Content type: release Language: ru Sources: [JetBrains RU]() Topics: [onnx](), [object-detection](), [Maven](), [Maven Central](), [Artificial Intelligence](), [Keras](), [PyTorch]() Tags: [big-data](), [central](), [deep-learning](), [fine-tuning](), [image-recognition](), [keras](), [kotlin](), [machine-learning](), [maven-central](), [object-detection](), [onnx](), [onnx-runtime](), [pytorch](), [tensorflow]() ### AI overview This release article presents KotlinDL 0.3, adding ONNX model support through ONNX Runtime Java API, new ModelHub models, an experimental high-level image-recognition API, additional layers, and Maven Central availability. It also describes object-detection support and fine-tuning limitations for ONNX models. ### Source excerpt Представляем версию 0.3 библиотеки глубокого обучения KotlinDL! Вас ждет множество новых фич: новые модели в ModelHub (включая модели для обнаружения объектов и распознавания лиц), возможность дообучать модели распознавания изображений, экспортированные из Keras и PyTorch в ONNX, экспериментальный высокоуровневый API для распознавания изображений и множество новых слоев, добавленных контрибьюторами. Также KotlinDL теперь доступен в Maven Central. В этой статье мы коснемся самых главных изменений релиза 0.3. Полный список изменений доступен по ссылке. Узнать больше о релизе ## Vespa Product Updates, December 2018 - ONNX Import and Map Attribute Grouping DevFeed: [Vespa Product Updates, December 2018 - ONNX Import and Map Attribute Grouping]() Original publisher: [Read original article]() Author: amberwilsonla-blog Published: 2018-12-13T22:18:41Z Content type: release Language: en Sources: [Yahoo]() Topics: [onnx](), [AI Models](), [API](), [ci](), [Continuous integration]() Tags: [cost](), [format](), [import](), [maps](), [onnx](), [streaming](), [tensorflow](), [tensors](), [yahoo-engineering]() ### AI overview This Vespa product update covers improved Streaming Search performance, ONNX model importing and conversion to Tensors for ranking, more precise transaction log pruning during restarts, and grouping on map attribute fields. ### Source excerpt yahoodevelopers: Today we're kicking off a blog post series of need-to-know updates on Vespa, summarizing the features and fixes detailed in Github issues. We welcome your contributions and feedback about any new features or improvements you'd like to see. For December, we're excited to share the following product news: Streaming Search Performance Improvement Streaming Search is a solution for applications where each query only searches a small, statically determined subset of the corpus. In this case, Vespa searches without building reverse indexes, reducing storage cost and making writes more efficient. With the latest changes, the document type is used to further limit data scanning, resulting in lower latencies and higher throughput. Read more here. ONNX Integration ONNX is an open ecosystem for interchangeable AI models. Vespa now supports importing models in the ONNX format and transforming the models into Tensors for use in ranking. This adds to the TensorFlow import included earlier this year and allows Vespa to support many training tools. While Vespa's strength is real-time model evaluation over large datasets, to get started using single data points, try the stateless model evaluation API. Explore this integration more in Ranking with ONNX models. Precise Transaction Log Pruning Vespa is built for large applications running continuous integration and deployment. This means nodes restart often for software upgrades, and node restart time matters. A common pattern is serving while restarting hosts one by one. Vespa has optimized transaction log pruning with prepareRestart, due to flushing as much as possible before stopping, which is quicker than replaying the same data after restarting. This feature is on by default. Learn more in live upgrade and prepareRestart. Grouping on Maps Grouping is used to implement faceting. Vespa has added support to group using map attribute fields, creating a group for values whose keys match the specified key, or field valu ## Why Curalate migrated from Caffe to MXNet for deep learning development and deployment DevFeed: [Why Curalate migrated from Caffe to MXNet for deep learning development and deployment]() Original publisher: [Read original article]() Published: 2018-08-01T00:00:00Z Content type: article Language: en Sources: [Curalate]() Topics: [Deep learning](), [Computer vision](), [Framework](), [Deployment](), [Development](), [Microservice](), [ImageNet](), [onnx]() Tags: [computer-vision](), [deep-learning](), [deeplearning](), [deployment](), [development](), [fork](), [framework](), [model](), [mxnet](), [onnx](), [python](), [scala]() ### AI overview Curalate describes migrating from Caffe to MXNet for computer vision systems. The post explains how MXNet supports experimentation, pre-trained models, model conversion, and integration into microservices and production environments. ### Source excerpt This post was simultaneously published to Medium. At Curalate, we use state of the art deep learning and computer vision to add a layer of magic to our products. Intelligent Product Tagging, for example, identifies our clients' products in user-generated photos. Being a startup, we need to build these deep learning and computer vision systems the same way we do the rest of our products: quickly. Our computer vision systems are built in two phases, research and productization, and we require a deep learning framework that accelerates both. During the research phase, we need a framework that's quick to get started with and is flexible enough to experiment with new ideas. Once we have a solution, we need a framework that can easily be integrated into a microservice and deployed to multiple production environments. In the past, we used Caffe for experimentation and our own custom inference interface to deploy the trained models to production. Experimentation was slow due to Caffe's dated Python API, lack of automatic differentiation, unreliable build/install process, and clunky support for advanced layers which required us to maintain our own custom fork. Productization of Caffe was challenging since we had to maintain our own JNI interface. We needed new and modern framework that fulfilled all of our needs while saving us from the shortcomings of Caffe. After a review of all the available options, we decided to move to MXNet. In this post, we'll discuss why we migrated to MXNet as our deep learning framework of choice to facilitate our speed of experimentation, development, and deployment. Training and Experimentation Whenever we are faced with a new computer vision problem, we start by looking at existing state-of-the-art implementations. If we are lucky the functionality of the service we are implementing is similar to an existing pre-trained model for MXNet. MXNet has a fairly fleshed out and maintained Model Zoo that contains all of the standard pre-trained models