# serverless inference

Published articles for serverless inference.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Roboflow Serverless vs. Self-Hosted Inference: What a 90-Second Traffic Spike Does to One GPU Server

DevFeed: [Roboflow Serverless vs. Self-Hosted Inference: What a 90-Second Traffic Spike Does to One GPU Server](<https://devfeed.tech/articles/roboflow-serverless-vs-self-hosted-inference-what-a-90-second-traffic-spike-does-to-one-gpu-server-124832.md>)

Original publisher: [Read original article](<https://roboflow.com/blog/roboflow-serverless-vs-self-hosted-inference-what-a-90-second-traffic-spike-does-to-one-gpu-server>)

Author: Sachin Agarwal

Published: 2026-11-09T18:16:06Z

Content type: comparison

Language: en

Sources: [Roboflow](<https://devfeed.tech/sources/roboflow-blog.md>)

Topics: [Serverless](<https://devfeed.tech/topics/serverless.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>)

Tags: [autoscaling](<https://devfeed.tech/tags/autoscaling.md>), [backlog](<https://devfeed.tech/tags/backlog.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [cameras](<https://devfeed.tech/tags/cameras.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [computer-vision](<https://devfeed.tech/tags/computer-vision.md>), [container](<https://devfeed.tech/tags/container.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [demand](<https://devfeed.tech/tags/demand.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [detection](<https://devfeed.tech/tags/detection.md>), [failure](<https://devfeed.tech/tags/failure.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [inference](<https://devfeed.tech/tags/inference.md>), [load](<https://devfeed.tech/tags/load.md>), [self-hosted](<https://devfeed.tech/tags/self-hosted.md>), [serverless](<https://devfeed.tech/tags/serverless.md>), [serverless-inference](<https://devfeed.tech/tags/serverless-inference.md>), [test](<https://devfeed.tech/tags/test.md>), [traffic](<https://devfeed.tech/tags/traffic.md>)

### AI overview

In a three-minute test of vehicle counting on 1280x1280 aerial frames, a single NVIDIA L40S self-hosted server handled 5 requests per second at about one-second median latency. At 20 requests per second for 90 seconds, its throughput leveled off around 13-15 requests per second; latency reached the 20-second timeout, and responses stopped even after traffic returned to baseline. Roboflow Serverless handled the same traffic, returning 2,273 successful requests at a 1.4-second median, versus 1,095 for self-hosting; reported failures were 0% rounded versus 50%. Each request triggered 16 tiled model inferences. The author interprets the self-hosted latency ramp as backlog growth, while noting the queue was not measured directly. The article says self-hosting can address bursts by provisioning peak capacity, with GPUs idle between bursts, or autoscaling; in this setup, bringing up a GPU server took about five minutes, longer than the burst. Roboflow Serverless is described as using the same open-source inference server, with per-model queues, warm spare capacity, and scaling on total backlog.

### Source excerpt

Sizing GPUs for the traffic you expect works until traffic does something you didn't expect. In our load test, a short burst left one GPU server timing out even after traffic returned to normal, while Roboflow Serverless kept up. Here's what went wrong, and why autoscaling can't add a GPU in time.

## GearUp: DAG-aware Acceleration for Serverless Inference Serving via Slack-guided Pre-loading

DevFeed: [GearUp: DAG-aware Acceleration for Serverless Inference Serving via Slack-guided Pre-loading](<https://devfeed.tech/articles/gearup-dag-aware-acceleration-for-serverless-inference-serving-via-slack-guided-pre-loading-107568.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.38196>)

Author: Qi Qi, Yifan Sui, Hanfei Yu, Hong Zhang, Hao Wang

Published: 2026-10-30T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [model-serving](<https://devfeed.tech/topics/model-serving.md>), [Serverless](<https://devfeed.tech/topics/serverless.md>), [lazy loading](<https://devfeed.tech/topics/lazy-loading.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [acceleration](<https://devfeed.tech/tags/acceleration.md>), [adaptive](<https://devfeed.tech/tags/adaptive.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-os](<https://devfeed.tech/tags/cs-os.md>), [dag](<https://devfeed.tech/tags/dag.md>), [inference](<https://devfeed.tech/tags/inference.md>), [loading](<https://devfeed.tech/tags/loading.md>), [serverless](<https://devfeed.tech/tags/serverless.md>), [serverless-inference](<https://devfeed.tech/tags/serverless-inference.md>), [serving](<https://devfeed.tech/tags/serving.md>), [slack](<https://devfeed.tech/tags/slack.md>)

### AI overview

GearUp hides machine-learning node initialization from a serverless inference DAG's critical path by profiling library, model-weight, and CUDA-kernel loading and scheduling it during upstream execution slack. It reuses initialized instances across requests and applies a probabilistic, traffic-adaptive keep-alive policy to trade idle memory use against cold-start risk. In tests on AMBER Alert and FINRA-ML with NVIDIA L40S GPUs, it cut median end-to-end latency by 64.5% and 75.9% versus SMIless, with median speedups of 3.1x and 10.0x over cold execution. Versus InstaInfer's eager pre-loading, integrated GPU and host memory allocation fell by up to 83.9% and 94.6%. The paper also reports 4.2x strong scaling on four workers.

### Source excerpt

arXiv:2610.38196v1 Announce Type: new Abstract: Serverless platforms increasingly host machine learning (ML) inference workflows, modeled as directed acyclic graphs (DAGs) of interdependent functions. Beyond container startup, each ML node must load libraries, model weights, and CUDA kernels before it can run. These costs accumulate along dependent DAG paths and place initialization on the end-to-end critical path. Existing systems optimize one half of this problem. DAG-agnostic pre-loading prepares nodes the request may never reach, wasting host and GPU memory. DAG-aware serving routes and scales whole functions, leaving the loading stages inside a node untouched. This paper presents GearUp, a serverless DAG inference serving system that removes ML-node loading from the end-to-end critical path at low resource cost. GearUp profiles the loading stages of each node and schedules them inside the slack time the DAG exposes, hiding initialization behind upstream execution. It reuses initialized instances within and across requests to avoid redundant cold starts. A probabilistic, traffic-adaptive keep-alive policy sets how long idle instances are retained, balancing memory cost against cold-start risk under bursty load. On the AMBER Alert and FINRA-ML workflows with NVIDIA L40S GPUs, GearUp reduces median end-to-end latency by 64.5% and 75.9% against SMIless, for median speedups of 3.1x and 10.0x over cold execution. Integrated GPU and host memory allocation fall by up to 83.9% and 94.6% against eager pre-loading in InstaInfer. GearUp achieves 4.2x strong scaling on four workers.

## Prompt Caching is GA on DigitalOcean Serverless Inference: Up to 90% Off the Context Your Agents Resend

DevFeed: [Prompt Caching is GA on DigitalOcean Serverless Inference: Up to 90% Off the Context Your Agents Resend](<https://devfeed.tech/articles/prompt-caching-is-ga-on-digitalocean-serverless-inference-up-to-90-off-the-context-your-agents-resend-97779.md>)

Original publisher: [Read original article](<https://www.digitalocean.com/blog/now-available-prompt-caching>)

Author: DigitalOcean

Published: 2026-10-23T16:48:34Z

Content type: release

Language: en

Sources: [DigitalOcean](<https://devfeed.tech/sources/digitalocean.md>)

Topics: [prompt](<https://devfeed.tech/topics/prompt.md>), [AI Context Management](<https://devfeed.tech/topics/ai-context-management.md>), [session affinity](<https://devfeed.tech/topics/session-affinity.md>), [context window](<https://devfeed.tech/topics/context-window.md>), [Serverless](<https://devfeed.tech/topics/serverless.md>)

Tags: [affinity](<https://devfeed.tech/tags/affinity.md>), [agents](<https://devfeed.tech/tags/agents.md>), [caching](<https://devfeed.tech/tags/caching.md>), [digitalocean](<https://devfeed.tech/tags/digitalocean.md>), [product-updates](<https://devfeed.tech/tags/product-updates.md>), [prompt-caching](<https://devfeed.tech/tags/prompt-caching.md>), [router](<https://devfeed.tech/tags/router.md>), [routing](<https://devfeed.tech/tags/routing.md>), [serverless-inference](<https://devfeed.tech/tags/serverless-inference.md>)

### AI overview

DigitalOcean says prompt caching is generally available for supported models in its open model catalog, with no request-code changes required. Cached input costs up to 90% less, and time to first token drops by up to 80%, though savings depend on the model and require a cache hit. Cache affinity steers requests to replicas holding most of the prompt, then balances across the least busy warm replicas; it requires no session ID. The publisher reports 85-95% cache hit rates on flagship open models in October 2026 testing. The excerpt ends mid-sentence, so it does not provide the full explanation or qualification of the feature.

### Source excerpt

Prompt caching is now generally available on DigitalOcean Serverless Inference, across supported models in our open model catalog, with no changes to your request code required. Cached input is billed at up to 90% off and time to first token drops by up to 80%.* An agent never starts over. Each new call adds the result of its last tool call to the end of a context that already holds the system prompt, the tool definitions, every file it has read, and every earlier action and result. On DigitalOcean Serverless Inference, more than 90% of the input tokens that agents send are tokens the model has already processed. Prompt caching cuts the cost of those tokens by up to 90% depending on the model, but only when the request reaches the replica holding the cache. Without cache-aware routing, a repeat request lands wherever the load balancer sends it, and on the wrong replica you pay full price for tokens you already sent. DigitalOcean cache affinity routes each request to the replica that already holds most of its prompt. There's no session ID to manage: a conversation's second message already contains its first, so the router finds the match on its own. On our flagship open models, cache hit rates measured in our October 2026 testing ran between 85 and 95%. How cache affinity makes hit rates reliable Your cached context lives on one specific replica. If your next request lands somewhere else, that replica doesn't have your cache, and you pay full price even though your prompt repeats. Serverless traffic spreads across many replicas, so without something steering repeat requests back to the right one, hit rate is mostly a coin flip. During public preview, caching was on for everyone and the discount was real, but whether you hit the cache was mostly chance. Cache affinity fixes that. For every request, the router checks which replicas already hold most of a request's prompt, narrows to that warm set, and only then balances load across whichever of those is least busy. If

## From training to serving: Reflection's Beam on CoreWeave

DevFeed: [From training to serving: Reflection's Beam on CoreWeave](<https://devfeed.tech/articles/from-training-to-serving-reflection-s-beam-on-coreweave-88754.md>)

Original publisher: [Read original article](<https://wf.coreweave.com/blog/from-training-to-serving-reflections-beam-on-coreweave>)

Author: CoreWeave

Published: 2026-10-16T21:07:03Z

Content type: article

Language: en

Sources: [Coreweave Blog Posts](<https://devfeed.tech/sources/coreweave-blog-posts.md>)

Topics: [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [Serverless](<https://devfeed.tech/topics/serverless.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [NVFP4](<https://devfeed.tech/topics/nvfp4.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [blackwell](<https://devfeed.tech/tags/blackwell.md>), [cluster](<https://devfeed.tech/tags/cluster.md>), [coding](<https://devfeed.tech/tags/coding.md>), [context-window](<https://devfeed.tech/tags/context-window.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [inference](<https://devfeed.tech/tags/inference.md>), [serverless](<https://devfeed.tech/tags/serverless.md>), [serverless-inference](<https://devfeed.tech/tags/serverless-inference.md>), [sft](<https://devfeed.tech/tags/sft.md>)

### AI overview

Reflection's open-weight Beam model targets software engineering and long-horizon, tool-driven workloads. The article describes its training and supervised fine-tuning on CoreWeave infrastructure, along with managed serverless inference access through an OpenAI-compatible API. It also recommends evaluating coding and agentic applications using task success, tool-use reliability, completion time, and cost per successful task.

### Source excerpt

Reflection's open-weight Beam model on CoreWeave spans training with Weights & Biases Models, SFT in partnership with Axolotl, and Day 0 API access on Serverless Inference in CoreWeave Forge.

## Cline Bot makes the case for an open AI coding harness

DevFeed: [Cline Bot makes the case for an open AI coding harness](<https://devfeed.tech/articles/cline-bot-makes-the-case-for-an-open-ai-coding-harness-66425.md>)

Original publisher: [Read original article](<https://siliconangle.com/2026/10/06/cline-makes-the-case-for-an-open-ai-coding-harness-fullyconnected/>)

Author: Sloane Kali Faye

Published: 2026-10-06T21:02:39Z

Content type: article

Language: en

Sources: [SiliconANGLE](<https://devfeed.tech/sources/siliconangle.md>)

Topics: [coding assistant](<https://devfeed.tech/topics/coding-assistant.md>), [Open Source Models & Datasets](<https://devfeed.tech/topics/open-source-models-datasets.md>), [DevInfra](<https://devfeed.tech/topics/devinfra.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-coding](<https://devfeed.tech/tags/ai-coding.md>), [ai-coding-harness](<https://devfeed.tech/tags/ai-coding-harness.md>), [cline](<https://devfeed.tech/tags/cline.md>), [cline-desktop](<https://devfeed.tech/tags/cline-desktop.md>), [coding](<https://devfeed.tech/tags/coding.md>), [coreweave](<https://devfeed.tech/tags/coreweave.md>), [cube-event-coverage](<https://devfeed.tech/tags/cube-event-coverage.md>), [dave-vellante](<https://devfeed.tech/tags/dave-vellante.md>), [fully-connected-2026](<https://devfeed.tech/tags/fully-connected-2026.md>), [fullyconnected](<https://devfeed.tech/tags/fullyconnected.md>), [john-furrier](<https://devfeed.tech/tags/john-furrier.md>), [model-training](<https://devfeed.tech/tags/model-training.md>), [nasdaq-crwv](<https://devfeed.tech/tags/nasdaq-crwv.md>), [news](<https://devfeed.tech/tags/news.md>), [open-weight-models](<https://devfeed.tech/tags/open-weight-models.md>), [renee-huang](<https://devfeed.tech/tags/renee-huang.md>), [serverless-inference](<https://devfeed.tech/tags/serverless-inference.md>), [software-development](<https://devfeed.tech/tags/software-development.md>), [thecube](<https://devfeed.tech/tags/thecube.md>)

### AI overview

Cline is making its AI coding harness open source and tuning it to support both frontier and smaller open-weight models. CEO Renee Huang says the company's token consumption grew 20-fold over four months, and describes expansion into general knowledge work through Cline Desktop and customizable tools for niche tasks.

### Source excerpt

Open-weight models keep multiplying, and the AI coding harness around them is becoming just as important to get right. Developers and enterprises are trading off cost against capability. The harness helps translate a model's capabilities into useful work. Cline Bot Inc. is betting that keeping that layer open will let more people build with AI. [...] The post Cline Bot makes the case for an open AI coding harness appeared first on SiliconANGLE.

## GLM-5.3-Flash launches on Bitdeer AI Model Studio

DevFeed: [GLM-5.3-Flash launches on Bitdeer AI Model Studio](<https://devfeed.tech/articles/mystery-solved-ox-alpha-is-glm-5-3-flash-and-it-s-now-live-on-bitdeer-ai-model-studio-87583.md>)

Original publisher: [Read original article](<https://www.bitdeer.ai/en/blog/glm-5-3-flash-live-on-bitdeer-ai/>)

Author: Evelyn Xiong

Published: 2026-08-27T11:48:00Z

Content type: article

Language: en

Sources: [Bitdeer AI Cloud](<https://devfeed.tech/sources/bitdeer-ai-cloud.md>)

Topics: [baseten](<https://devfeed.tech/topics/baseten.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Advanced AI workflows](<https://devfeed.tech/topics/advanced-ai-workflows.md>)

Tags: [agentic-systems](<https://devfeed.tech/tags/agentic-systems.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-applications](<https://devfeed.tech/tags/ai-applications.md>), [ai-model](<https://devfeed.tech/tags/ai-model.md>), [glm-5-3](<https://devfeed.tech/tags/glm-5-3.md>), [inference](<https://devfeed.tech/tags/inference.md>), [serverless-inference](<https://devfeed.tech/tags/serverless-inference.md>)

### AI overview

Bitdeer AI Model Studio now offers GLM-5.3-Flash through a serverless inference platform and OpenAI-compatible API. The article describes the model's 320B total parameters, 18B active parameters, native image and video input, and hybrid attention design for long contexts. It also outlines coding and agent use cases, deployment considerations, and performance figures reported by Z.ai.

### Source excerpt

Today, GLM-5.3-Flash is officially live on Bitdeer AI Model Studio, our serverless inference platform built for enterprise-grade security and elastic scaling.

## DigitalOcean Serverless Inference: A Deep Dive

DevFeed: [DigitalOcean Serverless Inference: A Deep Dive](<https://devfeed.tech/articles/digitalocean-serverless-inference-a-deep-dive-19943.md>)

Original publisher: [Read original article](<https://www.digitalocean.com/blog/serverless-inference-deep-dive>)

Author: smehta

Published: 2026-06-01T18:44:08Z

Content type: article

Language: en

Sources: [DigitalOcean](<https://devfeed.tech/sources/digitalocean.md>)

Topics: [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Serverless](<https://devfeed.tech/topics/serverless.md>), [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>), [hosting](<https://devfeed.tech/topics/hosting.md>), [Generative AI](<https://devfeed.tech/topics/generative-ai.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [anthropic](<https://devfeed.tech/tags/anthropic.md>), [api](<https://devfeed.tech/tags/api.md>), [api-key](<https://devfeed.tech/tags/api-key.md>), [audio](<https://devfeed.tech/tags/audio.md>), [budget](<https://devfeed.tech/tags/budget.md>), [chat](<https://devfeed.tech/tags/chat.md>), [code](<https://devfeed.tech/tags/code.md>), [context](<https://devfeed.tech/tags/context.md>), [cost](<https://devfeed.tech/tags/cost.md>), [deepseek](<https://devfeed.tech/tags/deepseek.md>), [digitalocean](<https://devfeed.tech/tags/digitalocean.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [image-generation](<https://devfeed.tech/tags/image-generation.md>), [inference](<https://devfeed.tech/tags/inference.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [key](<https://devfeed.tech/tags/key.md>), [models](<https://devfeed.tech/tags/models.md>), [serverless](<https://devfeed.tech/tags/serverless.md>), [serverless-inference](<https://devfeed.tech/tags/serverless-inference.md>)

### AI overview

DigitalOcean Serverless Inference provides managed access to 30+ models through one API key and endpoint, with automatic scaling and pay-per-token pricing without minimum commitments. It handles GPU allocation and model lifecycle; because it is sessionless, each request must include its full context. The API is OpenAI-compatible, and the excerpt says existing OpenAI code can be adapted by changing the base URL and key. Model Access Keys are recommended and can be scoped to models and restricted to a VPC.

### Source excerpt

The Problem: Inference Gets Hard at Scale If you've shipped an AI feature to production, you already know: the hard part isn't making a model respond to a prompt. The hard part is making it respond more reliably, at scale, across multiple models, without burning through your budget. The moment real users show up, you're dealing with GPU resource contention, traffic unpredictability (a single enterprise customer can 10x your request volume overnight), latency-cost tradeoffs that shift constantly, and multi-model orchestration across text, vision, image, video, and audio -- each with different API contracts and failure characteristics. Most teams spend months just getting the infrastructure stable. We built DigitalOcean Serverless Inference so you don't have to. What Serverless Inference Is DigitalOcean Serverless Inference is a fully managed, API-first inference platform -- 30+ foundation models across text, code, vision, image generation, video generation, and speech, all through a single API key, a single base URL, and pay-per-token pricing with no minimum commitments. The core idea: Serverless Inference separates model consumption from infrastructure management. It automatically scales to handle incoming requests. Because it does not maintain sessions, each request must include the full context needed by the model. You interact with models through an API surface. We handle GPU allocation, scaling, and model lifecycle underneath. Single Endpoint, Every Mode None https://inference.do-ai.run Authenticate with a Model Access Key (recommended -- scoped to specific models, VPC-restrictable) OpenAI and Anthropic Compatible The API is OpenAI-compatible. If you have existing code that calls OpenAI, switch to DigitalOcean by changing two lines -- the base URL and the key: Python from openai import OpenAI import os client = OpenAI( base_url="https://inference.do-ai.run/v1/", api_key=os.getenv("MODEL_ACCESS_KEY"), ) response = client.chat.completions.create( model="deepseek-v3.2"

## How we built the most performant DeepSeek V3.2, MiniMax-M2.5 and Qwen 3.5 397B on DigitalOcean Serverless Inference

DevFeed: [How we built the most performant DeepSeek V3.2, MiniMax-M2.5 and Qwen 3.5 397B on DigitalOcean Serverless Inference](<https://devfeed.tech/articles/how-we-built-the-most-performant-deepseek-v3-2-minimax-m2-5-and-qwen-3-5-397b-on-digitalocean-serverless-inference-19888.md>)

Original publisher: [Read original article](<https://www.digitalocean.com/blog/how-we-built-fastest-deepseek-minimax-qwen-on-blackwell-ultra>)

Author: Bhaskar Dutt

Published: 2026-04-28T09:00:00Z

Content type: article

Language: en

Sources: [DigitalOcean](<https://devfeed.tech/sources/digitalocean.md>)

Topics: [Inference](<https://devfeed.tech/topics/inference.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [performance-engineering](<https://devfeed.tech/topics/performance-engineering.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Serverless](<https://devfeed.tech/topics/serverless.md>), [qwen](<https://devfeed.tech/topics/qwen.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-native](<https://devfeed.tech/tags/ai-native.md>), [artificial-analysis](<https://devfeed.tech/tags/artificial-analysis.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [costs](<https://devfeed.tech/tags/costs.md>), [deepseek](<https://devfeed.tech/tags/deepseek.md>), [digitalocean](<https://devfeed.tech/tags/digitalocean.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [engine](<https://devfeed.tech/tags/engine.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [inference](<https://devfeed.tech/tags/inference.md>), [latency](<https://devfeed.tech/tags/latency.md>), [low-latency](<https://devfeed.tech/tags/low-latency.md>), [low-latency-inference](<https://devfeed.tech/tags/low-latency-inference.md>), [minimax](<https://devfeed.tech/tags/minimax.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [qwen](<https://devfeed.tech/tags/qwen.md>), [serverless](<https://devfeed.tech/tags/serverless.md>), [serverless-inference](<https://devfeed.tech/tags/serverless-inference.md>)

### AI overview

DigitalOcean says DeepSeek V3.2, MiniMax-M2.5, and Qwen 3.5 397B are generally available on its Serverless Inference service. It reports #1 output speed among providers tested by Artificial Analysis for DeepSeek V3.2 and Qwen 3.5 397B; for DeepSeek V3.2, it cites 230 output tokens per second and under one-second TTFT with 10,000 input tokens. The excerpt frames low latency as important for interactive and agentic workloads, where delays across sequential calls can accumulate, and says the post covers GPU work, serving-stack tuning, and technical tradeoffs. It provides no details of those optimizations.

### Source excerpt

Today at Deploy, we are announcing the general availability of DeepSeek V3.2, MiniMax-M2.5, and Qwen 3.5 397B on DigitalOcean Serverless Inference. On DeepSeek V3.2 and Qwen 3.5 397B, we deliver #1 output speed across all providers Artificial Analysis tested. On DeepSeek V3.2 specifically, that translates to 230 output tokens per second and sub-1-second Time-to-First-Token (TTFT) for 10,000 input tokens. This post covers how we got there: the GPU-level work, the serving stack tuning, and the specific technical tradeoffs we made along the way. Why fast inference matters The focus in AI development has fundamentally shifted from the training of models to the efficiency of inference. This shift is driven by the proliferation of agentic workloads, copilots, and real-time systems that form the core of next-generation AI applications. For these applications, speed is no longer just a performance metric; it is the critical differentiator between an engaging product and one that users abandon. Specifically, low-latency inference is essential for a seamless end-user experience. For highly interactive applications like conversational agents and voice interfaces, any delay beyond a sub-1-second TTFT is perceived as sluggish. The importance of fast inference is compounded by the complexity of modern AI workflows. An agentic task, for instance, often involves dozens of sequential model calls, where even minute Time-Per-Output-Token (TPOT) delays can accumulate into several seconds of user-visible latency. Quick inference also helps businesses by providing reliable performance and lower costs. Optimization in this area, such as that provided by DigitalOcean's inference engine, allows enterprises to achieve superior token economics, sustained throughput, and predictable latency, which are essential for scaling their AI-native applications reliably and affordably. Leading the Artificial Analysis benchmarks on speed The benchmarks we're publishing today reflect this. On DeepSeek V3.