# Grafana Labs blog on Grafana Labs Recent content in Grafana Labs blog on Grafana Labs This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Digital Experience Monitoring with Grafana Cloud: Session Replay, synthetic checks, and faster investigations DevFeed: [Digital Experience Monitoring with Grafana Cloud: Session Replay, synthetic checks, and faster investigations]() Original publisher: [Read original article]() Author: Bukola Ayodele Published: 2026-09-15T01:35:30.954353Z Content type: article Language: en Sources: [Grafana Labs blog on Grafana Labs]() Topics: [digital experience monitoring](), [Grafana Cloud](), [Frontend observability](), [synthetic monitoring](), [real user monitoring](), [session replay](), [Instrumentation](), [web applications](), [JavaScript](), [Core Web Vitals](), [Traces]() Tags: [analytics](), [core-web-vitals](), [digital-experience-monitoring](), [frontend-observability](), [grafana](), [grafana-cloud](), [grafana-cloud-frontend-observability](), [instrumentation](), [javascript](), [logs](), [metrics](), [monitoring](), [observability](), [real-user-monitoring](), [session-replay](), [synthetic](), [synthetic-monitoring](), [traces](), [web-applications]() ### AI overview This article explains how Digital Experience Monitoring in Grafana Cloud combines Frontend Observability, Synthetic Monitoring, real user monitoring, and session replay. It shows how these capabilities help engineering teams detect issues proactively, understand their scope and root causes, correlate frontend signals with backend traces, and resolve production problems faster. ### Source excerpt When something breaks in production, the questions that matter most are also the toughest to answer from metrics alone: who was affected, what did they actually see, and is this worth waking someone up for? Answering those questions requires a fuller picture of the issue and its impact on your users. That's where Digital Experience Monitoring (DEM) in Grafana Cloud comes in. By combining Frontend Observability and Synthetic Monitoring, DEM connects real user experiences with proactive testing, helping engineering teams understand the scope of an issue, investigate its cause, and resolve it faster, all within Grafana Cloud. In this blog post, we'll walk through some of the latest DEM updates in Grafana Cloud, and how to get started. You can also learn more by watching the video below. First, what is Digital Experience Monitoring? Digital Experience Monitoring in Grafana Cloud gives you a complete picture of how users experience your web applications, from real user data to proactive synthetic checks. DEM helps your team achieve: Real user visibility: know how users truly experience your web application, not just what your backend metrics suggest. Proactive detection: catch problems before your users do, using automated checks against your critical user journeys. End-to-end correlation: connect a frontend signal to the backend trace behind it. Faster resolution: cut your mean time to recovery from hours to minutes. Session Replay: see exactly what your users saw Session Replay in Grafana Cloud Frontend Observability lets you visually replay what a user saw and did inside your web application. Your team can watch exactly what users experienced and correlate it with real user monitoring signals like Core Web Vitals, user actions, and traces, which makes it a powerful tool for investigating bugs and running root cause analysis. Session Replay is powered by Faro, Grafana's open source JavaScript instrumentation library for collecting real user monitoring data. Let's walk ## How to monitor Cypress tests with Grafana Cloud DevFeed: [How to monitor Cypress tests with Grafana Cloud]() Original publisher: [Read original article]() Author: Rajesh Mahalingaswamy Published: 2026-09-12T11:22:06.456390Z Content type: tutorial Language: en Sources: [Grafana Labs blog on Grafana Labs]() Topics: [Grafana](), [Node.js](), [ci]() Tags: [analytics](), [cypress](), [grafana](), [grafana-alloy](), [grafana-cloud](), [grafana-cloud-metrics](), [how-to](), [logs](), [metrics](), [monitoring](), [node-js](), [observability](), [prometheus](), [traces]() ### AI overview A tutorial for monitoring Cypress test results by converting them to Prometheus metrics, pushing them through a Pushgateway, scraping them with Alloy, and forwarding them to Grafana Cloud Metrics. ### Source excerpt If your Cypress suite has tests that fail more often or run slower, you know it can be hard to figure out the pattern from a single job. It could be one spec that slowed down, or a single test that fails, or maybe the entire suite is trending slower. The root cause could be a bug in the app, or a flaky test, or something else. Your terminal output and CI log will tell you what happened on a single run, but that doesn't help you spot any larger trends--especially since you lose that data as soon as the job finishes. Thankfully, Cypress, a front-end automated test framework built for web applications, already exposes everything you need through its plugin hooks. After each spec finishes, Cypress hands you a results object with pass and fail counts, per-test durations, and states. You just need to turn that into metrics and ship it somewhere durable. In this post, you'll learn how to monitor your Cypress tests by converting those results into Prometheus metrics inside a Cypress hook, pushing them to a Prometheus Pushgateway, and letting Alloy scrape the gateway and forward everything to Grafana Cloud Metrics--using nothing but the free tiers. By the end you'll have a pipeline running with the following architecture: What you'll need This walkthrough runs everything alongside your existing Cypress project. Before you start, make sure you have: A Cypress project (this example uses Cypress 14.x) with a cypress.config.js you can edit A Prometheus Pushgateway. Cypress runs are short-lived batch jobs, so they can't be scraped directly--the Pushgateway holds the metrics between runs so a scraper can pick them up. Set up a Prometheus Pushgateway in your infrastructure Alloy, our open source collector we use to scrape the Pushgateway and remote-write to Grafana Cloud A Grafana Cloud account. The free tier includes Grafana Cloud Metrics and a Prometheus remote-write endpoint. If you don't have an account, you can sign up here Your Grafana Cloud remote-write URL, numeric user ID, an ## From failed check to real user impact: Pairing Synthetic Monitoring and Frontend Observability in Grafana Cloud DevFeed: [From failed check to real user impact: Pairing Synthetic Monitoring and Frontend Observability in Grafana Cloud]() Original publisher: [Read original article]() Author: Mark Meier Published: 2026-09-12T11:22:06.456390Z Content type: tutorial Language: en Sources: [Grafana Labs blog on Grafana Labs]() Topics: [synthetic monitoring](), [Frontend observability](), [Grafana Cloud](), [observability]() Tags: [analytics](), [frontend-observability](), [grafana](), [grafana-cloud](), [logs](), [metrics](), [monitoring](), [observability](), [synthetic-monitoring](), [traces](), [uptime]() ### AI overview This article explains how to combine Grafana Cloud Synthetic Monitoring with Frontend Observability to connect proactive checks with real-user impact. It describes the blind spots of synthetic-only monitoring and presents a combined workflow for faster triage, impact-aware alerts, and tests that evolve with real traffic. ### Source excerpt Say you get a support escalation about a page in the app that won't load. But when you pull up your synthetic checks, they're all green: 100% uptime, probes are passing. Something's not adding up, but which one do you trust? If you've run Grafana Cloud Synthetic Monitoring, you've been on both sides of this. Sometimes it's the ticket: real users hit a wall on the path but your checks pass cleanly. Other times, it's the inverse: a check is failing, you're in a panic, and you start trying to reproduce things for 30 minutes--only to find it was a blip from a single region, with minimal impact to real users. Neither the green dashboard nor the red alert were lying, they just weren't answering the correct question. This ends up being the root problem. Synthetic Monitoring is exceptionally good at telling you if something broke. It can not, however, tell you who it happened to, how bad it was, or why it matters. This is not a flaw in Synthetic Monitoring; it's the boundary of what a controlled, scheduled test can know. Grafana Cloud Frontend Observability helps to close this gap. Synthetic Monitoring gives you a proactive, outside-in signal; Frontend Observability gives you the real-user, inside-out signal. Together they form a closed loop: synthetic alerts end up getting some real user context, and real user data can make your synthetic tests smart. In this post, we'll look at why a synthetic-only strategy can leave blind spots, what Frontend Observability adds, and walk through practical workflows for running them together in Grafana Cloud. Along the way, you'll learn that the payoff is concrete: faster triage, alerts that carry blast-radius context, and a check suite that evolves with real traffic instead of aging against it. Green checks don't mean happy users Synthetic Monitoring is an active signal. You script a journey or declare a target, run it on a schedule from known probe locations, and in return get clean consistent results. This precise control of variables i ## How to measure and improve instrumentation quality for better full-stack observability DevFeed: [How to measure and improve instrumentation quality for better full-stack observability]() Original publisher: [Read original article]() Author: Arpit kumar Published: 2026-09-12T11:22:06.456390Z Content type: article Language: en Sources: [Grafana Labs blog on Grafana Labs]() Topics: [Instrumentation](), [observability](), [Grafana Cloud](), [telemetry]() Tags: [analytics](), [grafana](), [grafana-cloud](), [how-to](), [instrumentation](), [knowledge-graph](), [logs](), [metrics](), [monitoring](), [observability](), [traces]() ### AI overview This article explains how to measure and improve instrumentation quality for full-stack observability. It introduces Grafana Cloud's continuously updated instrumentation quality report, which evaluates each service using automated checks for telemetry completeness, correctness, signal coverage, naming, Kubernetes correlation, and connections across the service graph. ### Source excerpt Modern engineering teams instrument everything, with metrics, logs, traces, and profiles flowing from hundreds of services at once. But full-stack observability isn't really about collecting more telemetry; it's about having a single, unified picture of how your services connect to every layer beneath them, including their dependencies, the pods and nodes they run on, and the logs, traces, and profiles that explain their behavior. But there's often a quiet problem hiding underneath all that data: not all instrumentation is created equal, and every gap silently breaks one of those connections. One service, for example, might emit metrics but no logs, so when you pivot from "this is erroring" to "show me why," you hit a dead end. Another might have logs but an invalid service.name or a missing k8s.pod.name that breaks correlation, dropping it out of the graph and away from its pods and nodes. A third service might look perfectly healthy right up until an incident, when you discover its traces were never wired up and the trail goes cold exactly when you need it most. To fix this, Grafana Cloud's Knowledge Graph now includes an instrumentation quality report: an automated, continuously updated assessment of how well each of your services is instrumented--and, in effect, how they plug into the full-stack picture. In this post, we'll walk through how to read the instrumentation quality report, how the scoring works, and how to use it to systematically raise the observability bar, keeping every layer of your stack joined up across every service you run. What is instrumentation quality? Instrumentation quality is a measure of how complete and correct the telemetry for a given service is, judged against a set of automated checks. Each service is evaluated by a server-computed set of quality checks: small, focused rules that validate one specific thing about a service's telemetry. A few examples: Does the service emit logs? Are service graph metrics present? Is the service nam ## Better context, smarter testing: How to give your AI coding agent direct access to k6 docs DevFeed: [Better context, smarter testing: How to give your AI coding agent direct access to k6 docs]() Original publisher: [Read original article]() Author: İnanç Gümüş Published: 2026-09-12T11:22:06.456390Z Content type: article Language: en Sources: [Grafana Labs blog on Grafana Labs]() Topics: [k6](), [Agent Skill](), [AI Agent](), [Command-line interface](), [coding]() Tags: [agent-skill](), [ai](), [ai-agent](), [ai-coding](), [analytics](), [cli](), [grafana](), [k6](), [logs](), [metrics](), [monitoring](), [observability](), [performance-testing](), [testing](), [traces]() ### AI overview The article introduces k6 x docs, an official k6 2.0 command that provides k6 documentation directly in the terminal. It supports API references, guides, best practices, and examples; works offline after the first lookup; matches documentation to the installed k6 version; and includes an agent skill for AI coding assistants. ### Source excerpt As testing workflows become more AI-assisted, fast access to accurate documentation matters more than ever. Whether you're writing a new load test, troubleshooting an issue, or having an AI agent generate a script for you, you need reliable guidance that keeps pace with the way you work. But most documentation still lives in a browser. Every time you or your agent needs to verify an API or look up a best practice, you're forced to leave your terminal or editor and interrupt your workflow. That's why, in k6 2.0, we introduced k6 x docs, an official k6 command that puts the entire k6 documentation library, including API references, guides, best practices, and examples, directly in your terminal. It works offline after first use, matches the docs to your exact k6 version, and includes a built-in agent skill, so AI coding assistants can look up k6 docs faster and most cost-effectively. The problem: documentation lives in the wrong place Most developers know the feeling. You're writing a k6 script, you need to check the signature for http.post or remember how thresholds work, and suddenly you're in a browser tab, searching, scrolling, clicking through navigation, and losing the context you had in your editor. For AI agents, the problem is worse. When they need to reference k6 APIs, best practices, or examples, they either rely on stale training data, hallucinate a function signature, or burn expensive tokens on a web search that may not return the right version of the docs. k6 x docs solves both problems by making documentation a first-class part of the k6 CLI. It gives agents CLI access to k6 docs that automatically detect the k6 version in use and deliver accurate content without leaving the session or performing web searches. How k6 x docs works The use is simple: type k6 x docs, optionally followed by the topic you want to look up. k6 x docs # See all available topics k6 x docs http # Learn about the k6/http module k6 x docs http get # Look up a specific function k6 ## Visual playback of the user journey: Introducing Session Replay in Grafana Cloud Frontend Observability DevFeed: [Visual playback of the user journey: Introducing Session Replay in Grafana Cloud Frontend Observability]() Original publisher: [Read original article]() Author: Lukasz Gut Published: 2026-09-12T11:22:06.456390Z Content type: article Language: en Sources: [Grafana Labs blog on Grafana Labs]() Topics: [session replay](), [Grafana Cloud Frontend Observability](), [Frontend observability](), [web applications](), [telemetry](), [SDKs]() Tags: [analytics](), [frontend-observability](), [grafana](), [grafana-cloud-frontend-observability](), [instrumentation](), [logs](), [metrics](), [monitoring](), [observability](), [privacy](), [session-replay](), [telemetry](), [traces](), [web-applications]() ### AI overview The article introduces Session Replay, a public-preview add-on for Grafana Cloud Frontend Observability. It visually reconstructs how a web application appeared and changed during a user session, linking that playback to session timelines and technical telemetry so engineering teams can investigate frontend problems more directly. ### Source excerpt Grafana Cloud Frontend Observability helps engineering teams quantify the end user experience by bringing metrics, logs, traces, and user session context to client-side web applications. Teams can monitor application health and performance over time, triage errors, and correlate frontend signals with backend telemetry to investigate issues across the stack. Yet some of the hardest frontend problems remain difficult to diagnose. A support ticket might report that a checkout button did nothing, a form unexpectedly reset, or a workflow broke only in one browser or on one device. Metrics can reveal a performance regression, logs can capture an error, and traces can expose a slow request, but no single signal shows what the interface actually looked like to the user. This is exactly why we built Session Replay, an add-on feature in Frontend Observability that provides a visual reconstruction of a user's journey, connected to the telemetry Grafana Cloud already collects. It helps engineering teams move from a reported problem to seeing what happened and knowing exactly where to investigate next. What is Session Replay in Grafana Cloud Frontend Observability? Session Replay, now in public preview, adds visual playback capabilities to Frontend Observability. It reconstructs how a web application appeared and changed as a user navigated and interacted with it, so you can observe the journey as it unfolded instead of inferring it from individual telemetry events. Frontend Observability already brings together a chronological timeline of the events within a user session. That timeline tells you what happened and when. Session Replay adds the missing visual context: what was happening in the interface around those events and how one interaction led to the next. Because both views belong to the same session, you can move between the user experience and the relevant technical signals without losing the thread of your investigation. Session Replay builds on the Grafana Faro Web SD ## Grafana 13.2 release: easier ways to query and explore your data DevFeed: [Grafana 13.2 release: easier ways to query and explore your data]() Original publisher: [Read original article]() Author: Grafana Labs Team Published: 2026-09-12T11:22:06.456390Z Content type: release Language: en Sources: [Grafana Labs blog on Grafana Labs]() Topics: [Grafana](), [dashboards]() Tags: [analytics](), [dashboards](), [enterprise](), [explore](), [grafana](), [grafana-cloud](), [logs](), [metrics](), [monitoring](), [observability](), [release](), [sql](), [traces]() ### AI overview Grafana 13.2 introduces generally available saved queries for Grafana Cloud and Grafana Enterprise, letting organizations store, discover, and reuse vetted queries across dashboards, Explore, and annotation queries. The release also highlights a new View panel sidebar for exploring busy panels. ### Source excerpt Grafana 13.2 is here, bringing more improvements to help you and your team explore your data and get to insights faster. In this post, we'll highlight the latest updates to saved queries, a feature that lets teams share, discover, and reuse queries to get to trusted answers faster and help new teammates get up to speed. We'll also explore how the new View panel sidebar makes exploring busy panels a breeze. If you want to read about all the latest updates in Grafana 13.2, please refer to the changelog or our What's New documentation. Saved queries: reuse trusted queries across dashboards and teams Good queries are hard-won. Writing one means knowing both the query language and your own data, like which of four similarly named metrics is the one you can trust. That knowledge usually sits with a few experienced people, or is gradually learned through exploration (increasingly AI-assisted), validation, and revision. Often teams end up rebuilding the same Grafana queries over and over, and the best ones live in pinned Slack messages or get copy-pasted from old dashboards. New team members feel it most, since their first weeks are often spent reverse-engineering existing dashboards just to work out how to ask a question of their own. The query history in Grafana Explore helps, keeping a couple of weeks of your own queries and letting you "star" the keepers. It's private to you, though. Until recently, there hasn't been a built-in way to take a query you trust and put it somewhere your whole organization can find it. How teams use saved queries We built saved queries, which is now generally available in Grafana Cloud and Grafana Enterprise, to address this challenge by providing a shared query library for your organization. When you write a query worth keeping, you can save it with a title, description, and tags. Saving works from dashboard panels, Explore, and annotation queries. This means teammates who don't know PromQL or SQL can still build dashboards from queries tha ## Knowledge Graph as context for LLMs: demonstrating decisive RCA and faster production performance DevFeed: [Knowledge Graph as context for LLMs: demonstrating decisive RCA and faster production performance]() Original publisher: [Read original article]() Author: Sarah Constant Published: 2026-09-12T11:22:06.456390Z Content type: article Language: en Sources: [Grafana Labs blog on Grafana Labs]() Topics: [Large Language Model](), [Grafana Cloud](), [debugging](), [incident](), [telemetry](), [observability ai agents]() Tags: [ai](), [ai-agents](), [analytics](), [debugging](), [grafana](), [grafana-cloud](), [incident](), [knowledge-graph](), [llms](), [logs](), [metrics](), [monitoring](), [observability](), [telemetry](), [traces]() ### AI overview The article reports Grafana Labs experiments testing whether providing AI agents with Grafana Cloud's Knowledge Graph improves incident debugging compared with raw telemetry alone. In one incident replayed 16 times per approach, Knowledge Graph context led to the correct root cause 15 times, versus once with raw telemetry. The article also describes challenges including misleading signals, unsupported confident answers, and inconsistent investigations, arguing that well-structured context matters more than context-window size alone. ### Source excerpt On the product team here at Grafana Labs, we consider AI agents our users, too. That's why we set out to test how well agents can debug incidents across the full stack, and how much better they perform with Grafana Cloud's Knowledge Graph vs. using raw telemetry alone. Our early results are promising. In one real incident we replayed 16 times each way, an agent with Knowledge Graph context found the correct root cause 15 times, compared with just once using raw telemetry alone. Along the way, we also uncovered some of the challenges that still stand in the way of reliable AI-assisted debugging, from chasing the wrong signals to confidently making things up and producing inconsistent answers. We're still early, but our findings point to an important idea. The industry's shorthand right now is that a bigger context window will lead to better outputs. Our findings suggest it's not just about more context; it's about structuring your data well enough to serve the right context. Here's a look at what we've learned so far, as we continue to experiment out in the open and bring you along, the Grafana Labs way. Giving an agent access to telemetry is just the beginning Give a current-generation model like Opus 4.8 access to your raw telemetry during a live incident, and it genuinely starts to figure things out: querying metrics and logs, forming a hypothesis, and checking it. We have watched it work on our own incidents, and it does it affordably. But if you run software at scale, where uptime is business-critical and large teams share the responsibility, a better model alone doesn't get you all the way there for debugging. From analyzing how LLMs do root-cause analysis on our own infrastructure, and from speaking to our customers, we've uncovered three problems that get in the way: The further the cause is from the alert, the more likely the model is to get it wrong. An agent may confidently make things up when it doesn't have the evidence it needs. The same investigation c ## Custom labels in Grafana Cloud Synthetic Monitoring: New updates for consistency and ease-of-use DevFeed: [Custom labels in Grafana Cloud Synthetic Monitoring: New updates for consistency and ease-of-use]() Original publisher: [Read original article]() Author: Anant Sharma Published: 2026-09-12T11:22:06.456390Z Content type: article Language: en Sources: [Grafana Labs blog on Grafana Labs]() Topics: [Grafana Cloud](), [synthetic monitoring](), [Grafana](), [Monitoring](), [telemetry](), [dashboards]() Tags: [analytics](), [dashboards](), [grafana](), [grafana-cloud](), [how-to](), [logs](), [metrics](), [monitoring](), [observability](), [synthetic-monitoring](), [telemetry](), [traces](), [update]() ### AI overview Grafana Cloud Synthetic Monitoring is updating custom labels so they attach directly to every check metric and log, rather than only sm_check_info. The label_ prefix will be removed, and labels will appear exactly as defined. Existing users must migrate dashboards, SLOs, alerts, and queries that reference these labels by March 1, 2027. ### Source excerpt Labels are a powerful way to organize telemetry and define policies across Grafana Cloud, helping to streamline alerting, attribution, access control, and more. But traditionally, custom labels in Synthetic Monitoring have worked a little differently: they only lived on a single sm_check_info metric, and Grafana Cloud prefixed each one with label_. To make custom labels in Synthetic Monitoring work consistently with the rest of Grafana Cloud--without extra joins, naming conventions, or workarounds--we're rolling out an update that lets your custom labels attach directly to every check metric, not just sm_check_info, and removes the label_ prefix. Starting today, labels appear exactly as you write them, making Synthetic Monitoring data easier to navigate and use with label-based policies across Grafana Cloud. If you currently use custom labels in Synthetic Monitoring, read on to learn how to migrate to the new labels. We are asking users to migrate by March 1, 2027 to ensure their custom dashboards, SLOs, alerts, and queries that reference Synthetic Monitoring metrics do not break, and continue to work as expected. If you do not use custom labels in Synthetic Monitoring, you don't need to do anything to prepare for this update. How custom labels work in Synthetic Monitoring Until now, if you wanted to filter a dashboard, scope an alert, or attribute cost by team or service within Synthetic Monitoring, you had to join sm_check_info against the check metric you actually want to query. You also had to remember that team is really label_team in this context. That approach worked to ensure your custom labels were never at odds with system-defined labels. However, it broke down as usage scaled up and dozens of teams started running hundreds of checks across services, environments, and regions. Teams rely on consistent schemas to direct label-based workflows, and this update brings Synthetic Monitoring further into the fold of your existing policies. With the update, labels i ## The Grafana AI SDK for Go: a shared foundation for building AI applications DevFeed: [The Grafana AI SDK for Go: a shared foundation for building AI applications]() Original publisher: [Read original article]() Author: Luccas Quadros Published: 2026-09-12T11:22:06.456390Z Content type: article Language: en Sources: [Grafana Labs blog on Grafana Labs]() Topics: [vercel ai sdk](), [SDKs](), [Go Language](), [AI Development](), [Large Language Model](), [agent observability](), [Model Context Protocol](), [React]() Tags: [agent-observability](), [ai](), [analytics](), [backend](), [building](), [go](), [grafana](), [logs](), [metrics](), [monitoring](), [observability](), [sdk](), [streaming](), [tools](), [traces]() ### AI overview Grafana Labs introduces the Grafana AI SDK for Go, an open-source shared foundation for building AI applications. The SDK standardizes model calls, streaming, tool execution, structured output, multi-step agents, workflow controls, and operational features such as retries, logging, metrics, and Agent Observability. It also supports streaming Go backends to Vercel AI SDK frontend hooks. ### Source excerpt Starting an experiment with an LLM has never been easier. Keeping a growing collection of those experiments consistent is another matter. Earlier this year, as more teams began exploring AI features here at Grafana Labs, we repeatedly encountered the same pattern: a new experiment would start, move quickly, and build its own client for whichever model provider it needed. The next experiment would do the same, with a slightly different abstraction for streaming, tools, errors, or provider configuration. This was understandable, given the circumstances. Model providers were changing quickly, our teams were learning quickly, and coding agents made it possible to turn an idea into a working integration faster than ever. But that speed also made it easier for every integration to develop its own architecture. Eventually, we were maintaining a collection of solutions to what was essentially the same problem. And since most of our backend is written in Go, we built the Grafana AI SDK for Go to give our teams a shared foundation to work from. It provides common interfaces for calling models, streaming responses, executing tools, producing structured output, and running multi-step agents. It also speaks the protocol used by Vercel AI SDK frontend hooks, so a Go backend can stream directly to useChat, useCompletion, and useObject. We built it because we needed it inside Grafana Labs, but we open sourced it last month (alongside a broader collection of tools we released for building, operating, and understanding AI systems during our first Grafana Labs AI Week) because we think other teams building AI applications in Go are likely to encounter many of the same problems. We would like to build the next part together, so in this blog I'll tell you a bit more about the project, including how you can put it to use today, as well as how you can help us improve it. What teams can build with it today The SDK supports both simple model calls and larger application workflows: Generate ## How to scale Alloy as a central telemetry gateway: capacity planning, load testing, and production lessons DevFeed: [How to scale Alloy as a central telemetry gateway: capacity planning, load testing, and production lessons]() Original publisher: [Read original article]() Author: Fatjon Nebiu Published: 2026-09-12T11:22:06.456390Z Content type: tutorial Language: en Sources: [Grafana Labs blog on Grafana Labs]() Topics: [Grafana]() Tags: [analytics](), [auth](), [best-practices](), [cloud](), [cost](), [data](), [deployment](), [enterprise](), [grafana](), [grafana-alloy](), [grafana-cloud](), [how-to](), [kubernetes](), [logs](), [metrics](), [monitoring](), [observability](), [opentelemetry](), [platform](), [production](), [prometheus](), [scale](), [telemetry](), [testing](), [traces]() ### AI overview A practical guide to scaling Grafana Alloy as a centralized telemetry gateway. It covers capacity planning, load testing, and production considerations for collecting metrics, logs, and traces and forwarding them to Grafana Cloud. ### Source excerpt Running Alloy as a single-instance sidecar is simple. Running it as a centralized gateway that absorbs the full telemetry stream of an enterprise platform--tens of millions of active series, terabytes of logs per day, and tens of thousands of trace spans per second--is a different challenge altogether. To get it right, you need deliberate capacity planning, honest load testing, and a monitoring setup that doesn't rely on the very thing you're testing. As part of the Professional Services team here at Grafana Labs, we've seen this firsthand working with customers. In this post, we'll walk you through the best practices we follow to help them find success, and we'll do so using real, anonymized data from a recent engagement. We'll cover how we sized and load tested a production Alloy central collector deployment on Kubernetes, what the numbers looked like under real stress, and how the cluster behaves today handling the full production telemetry workload for a large enterprise platform. By the end, you should have a better sense for how you can create your own central gateway for collecting telemetry in Grafana Cloud. Why a central gateway? Before diving into numbers, it's worth explaining the pattern. In a central gateway setup, all telemetry from application teams--metrics, logs, and traces--flows to a shared Alloy fleet via OTLP or native Prometheus/Loki write protocols. Alloy buffers, processes, batches, and forwards everything to Grafana Cloud. This gives you several things that per-team sidecar deployments struggle to provide: A single control plane: Auth, rate limiting, and routing in one place so application teams don't need to manage Grafana Cloud credentials Centralized buffering: Ensure a transient Grafana Cloud slowdown doesn't immediately cause data loss at the source Cost visibility: Configure the gateway to only accept telemetry data containing the label or attribute that is mandatory for cost-attribution Protocol normalization: Send OTLP, Prometheus Remote