# tracing

Tracing is an observability technique for representing the path of a request through an application as correlated spans.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Agent Observability Example: Tracing One Bad Answer to Its Root Cause

DevFeed: [Agent Observability Example: Tracing One Bad Answer to Its Root Cause](<https://devfeed.tech/articles/agent-observability-example-tracing-one-bad-answer-to-its-root-cause-146867.md>)

Original publisher: [Read original article](<https://montecarlo.ai/blog-agent-observability-example>)

Author: Virna Sekuj

Published: 2026-11-20T19:19:28Z

Content type: tutorial

Language: en

Sources: [Monte Carlo](<https://devfeed.tech/sources/monte-carlo.md>)

Topics: [agent observability](<https://devfeed.tech/topics/agent-observability.md>), [tracing](<https://devfeed.tech/topics/tracing.md>), [OpenTelemetry](<https://devfeed.tech/topics/opentelemetry.md>)

Tags: [agent-observability](<https://devfeed.tech/tags/agent-observability.md>), [agent-trace-span-example](<https://devfeed.tech/tags/agent-trace-span-example.md>), [agent-tracing-example](<https://devfeed.tech/tags/agent-tracing-example.md>), [ai-agent-observability-example](<https://devfeed.tech/tags/ai-agent-observability-example.md>), [ai-observability](<https://devfeed.tech/tags/ai-observability.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [context](<https://devfeed.tech/tags/context.md>), [context-layer](<https://devfeed.tech/tags/context-layer.md>), [example](<https://devfeed.tech/tags/example.md>), [llm-agent-monitoring-example](<https://devfeed.tech/tags/llm-agent-monitoring-example.md>), [performance](<https://devfeed.tech/tags/performance.md>), [root-cause](<https://devfeed.tech/tags/root-cause.md>), [tracing](<https://devfeed.tech/tags/tracing.md>), [what-to-monitor-in-ai-agents](<https://devfeed.tech/tags/what-to-monitor-in-ai-agents.md>)

### AI overview

In the hypothetical billing-agent case, customers got a 30-day refund answer because the vector index served an outdated policy, even though latency, tool behavior, and the judge's faithfulness score looked healthy. Tracing retrieval lineage upstream revealed that a CMS field rename caused the nightly embedding job to write zero rows while reporting success, freezing the index on policy v7. The described resolution updates the job, backfills missed content, adds freshness, volume, and schema-change monitoring linked through lineage, then replays affected questions to verify the agent cites v8 and answers "14 days." The example recommends monitoring context, performance, behavior, and output, and notes that answer evaluations inherit the blind spots of retrieved data.

### Source excerpt

A step-by-step agent observability example: how one bad answer was traced across context, performance, behavior, and output to its root cause, and fixed.

## Datadog LLM Observability: Features, Pricing, and Alternatives

DevFeed: [Datadog LLM Observability: Features, Pricing, and Alternatives](<https://devfeed.tech/articles/datadog-llm-observability-features-pricing-and-alternatives-145962.md>)

Original publisher: [Read original article](<https://newrelic.com/blog/ai/datadog-llm-observability>)

Author: John Blust

Published: 2026-11-20T11:51:52.599709Z

Content type: article

Language: en

Sources: [New Relic](<https://devfeed.tech/sources/new-relic.md>)

Topics: [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [datadog](<https://devfeed.tech/topics/datadog.md>), [agent observability](<https://devfeed.tech/topics/agent-observability.md>), [ai observability](<https://devfeed.tech/topics/ai-observability.md>), [observability](<https://devfeed.tech/topics/observability.md>), [tracing](<https://devfeed.tech/topics/tracing.md>), [OpenTelemetry](<https://devfeed.tech/topics/opentelemetry.md>), [LLM security](<https://devfeed.tech/topics/llm-security.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [phoenix](<https://devfeed.tech/topics/phoenix.md>), [LangChain](<https://devfeed.tech/topics/langchain.md>), [dynatrace](<https://devfeed.tech/topics/dynatrace.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-monitoring](<https://devfeed.tech/tags/ai-monitoring.md>), [alternatives](<https://devfeed.tech/tags/alternatives.md>), [compare](<https://devfeed.tech/tags/compare.md>), [datadog](<https://devfeed.tech/tags/datadog.md>), [devops](<https://devfeed.tech/tags/devops.md>), [evaluations](<https://devfeed.tech/tags/evaluations.md>), [features](<https://devfeed.tech/tags/features.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-observability](<https://devfeed.tech/tags/llm-observability.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [new-relic](<https://devfeed.tech/tags/new-relic.md>), [observability](<https://devfeed.tech/tags/observability.md>), [pricing](<https://devfeed.tech/tags/pricing.md>), [tracing](<https://devfeed.tech/tags/tracing.md>)

### AI overview

Datadog LLM Observability, now called Agent Observability, traces model calls and agent workflows, evaluates quality and safety, supports prompt experiments, and tracks latency, tokens, errors, and costs. It is a standalone product billed by model-call spans; tool, retrieval, and workflow spans do not count. Both listed plans include core tracing and evaluation features, and annual Pro pricing starts at $160 per month, with retention add-ons affecting per-span rates. Alternatives differ in approach: New Relic connects AI telemetry to existing APM, including optional eBPF monitoring; Langfuse offers a self-hostable core; LangSmith emphasizes agent tracing and evaluation workflows; and Arize Phoenix supports local deployment with OpenTelemetry-based instrumentation. The text notes that Datadog can correlate with its other products through shared trace IDs and supports Python, Node.js, Java, OpenTelemetry, and HTTP API integration.

### Source excerpt

An objective look at Datadog LLM Observability: tracing, evaluations, how pricing works, and how alternatives like New Relic AI Monitoring compare.

## AWS Lambda Durable Functions observability with OpenTelemetry

DevFeed: [AWS Lambda Durable Functions observability with OpenTelemetry](<https://devfeed.tech/articles/aws-lambda-durable-functions-observability-with-opentelemetry-144754.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/compute/aws-lambda-durable-functions-observability-with-opentelemetry/>)

Author: Ricardo Marques

Published: 2026-11-19T19:09:43Z

Content type: tutorial

Language: en

Sources: [AWS Compute Blog](<https://devfeed.tech/sources/aws-compute-blog.md>)

Topics: [OpenTelemetry](<https://devfeed.tech/topics/opentelemetry.md>), [tracing](<https://devfeed.tech/topics/tracing.md>), [observability](<https://devfeed.tech/topics/observability.md>), [serverless monitoring](<https://devfeed.tech/topics/serverless-monitoring.md>), [function as a service](<https://devfeed.tech/topics/function-as-a-service.md>), [Amazon CloudWatch](<https://devfeed.tech/topics/amazon-cloudwatch.md>), [Amazon CloudWatch Logs](<https://devfeed.tech/topics/amazon-cloudwatch-logs.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-lambda](<https://devfeed.tech/tags/aws-lambda.md>), [best-practices](<https://devfeed.tech/tags/best-practices.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [durable-execution](<https://devfeed.tech/tags/durable-execution.md>), [durable-functions](<https://devfeed.tech/tags/durable-functions.md>), [observability](<https://devfeed.tech/tags/observability.md>), [opentelemetry](<https://devfeed.tech/tags/opentelemetry.md>), [plugin](<https://devfeed.tech/tags/plugin.md>), [sdk](<https://devfeed.tech/tags/sdk.md>), [span](<https://devfeed.tech/tags/span.md>), [trace](<https://devfeed.tech/tags/trace.md>)

### AI overview

The `aws-durable-execution-sdk-python-otel` plugin assigns deterministic trace IDs so invocations of one AWS Lambda Durable Functions execution can appear as a unified trace, with spans reflecting operation wall-clock time and avoiding duplicate spans during replay. Setup requires Python 3.11+, the ADOT Lambda layer, X-Ray active tracing and write permissions, and passing `InvocationOtelPlugin` to the durable execution handler. The post distinguishes execution view, which groups work under the workflow span but depends on the vendor's span-ingestion window, from invocation view, which can export shorter invocation spans sooner; the final workflow span still covers the full execution and may be rejected if too old. Logger enrichment adds trace and span IDs to logs by default. The ADOT layer adds cold-start and memory overhead (about 40-60 MB); the post recommends measuring impact, naming durable operations, and using sampling to manage trace costs. The excerpt's walkthrough covers a sequential workflow with a human approval callback, not every SDK primitive or failure path.

### Source excerpt

A single AWS Lambda Durable Functions workflow can span many invocations, fragmenting its traces. This post shows how the OpenTelemetry plugin for the AWS Durable Execution SDK generates deterministic trace IDs that unify every invocation into one distributed trace, and how to set it up, read the traces, and apply best practices.

## Model Discovery for Virtual Chaos Engineering with Incomplete Tracing: An eBPF Study

DevFeed: [Model Discovery for Virtual Chaos Engineering with Incomplete Tracing: An eBPF Study](<https://devfeed.tech/articles/model-discovery-for-virtual-chaos-engineering-with-incomplete-tracing-an-ebpf-study-142173.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23313>)

Author: Anatoly A. Krasnovsky

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Chaos Engineering](<https://devfeed.tech/topics/chaos-engineering.md>), [eBPF](<https://devfeed.tech/topics/ebpf.md>), [tracing](<https://devfeed.tech/topics/tracing.md>), [Software Engineering](<https://devfeed.tech/topics/software-engineering.md>)

Tags: [association](<https://devfeed.tech/tags/association.md>), [berkeley-packet-filter](<https://devfeed.tech/tags/berkeley-packet-filter.md>), [case-study](<https://devfeed.tech/tags/case-study.md>), [chaos-engineering](<https://devfeed.tech/tags/chaos-engineering.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [dependencies](<https://devfeed.tech/tags/dependencies.md>), [discovery](<https://devfeed.tech/tags/discovery.md>), [ebpf](<https://devfeed.tech/tags/ebpf.md>), [instrumentation](<https://devfeed.tech/tags/instrumentation.md>), [legacy-systems](<https://devfeed.tech/tags/legacy-systems.md>), [model](<https://devfeed.tech/tags/model.md>), [opentelemetry](<https://devfeed.tech/tags/opentelemetry.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [protocols](<https://devfeed.tech/tags/protocols.md>), [semantics](<https://devfeed.tech/tags/semantics.md>), [study](<https://devfeed.tech/tags/study.md>), [tracing](<https://devfeed.tech/tags/tracing.md>), [virtual](<https://devfeed.tech/tags/virtual.md>)

### AI overview

The study finds that eBPF observation can support virtual fault experiments in some tested applications even when application SDK tracing is absent. Across 108 cases in three public applications, the researchers combined discovered dependencies with declared operation requirements, deployment facts, and scenario assumptions, then compared model forecasts with independent execution checks. With SDK tracing disabled, the tested OpenTelemetry eBPF Instrumentation path supported all seven operations in two applications over three repetitions; forecasts matched exactly in 17 of 21 operation-repetition groups, with a maximum state-weighted discrepancy of 4.22 percentage points. Adding observed context links supported nine more groups under partial tracing. The acquisition path remains limited by protocols, evidence association, and operation semantics.

### Source excerpt

arXiv:2611.23313v1 Announce Type: new Abstract: Model-based fault experiments can assess the consequences of component failures before intervening in a running application. Recent work on model discovery and graph simulation acquired operation models from application telemetry and used them for availability prediction. Applying this approach to operational legacy systems raises an acquisition problem: the required traces may be absent or incomplete, while introducing them can require changes across heterogeneous components. These systems continue to serve business operations during extended maintenance and modernization. External observation offers a possible route to model-based fault analysis before a tracing retrofit, but collecting telemetry does not by itself establish that an executable operation model can be obtained. We investigate this route using extended Berkeley Packet Filter (eBPF) observations and the existing graph-based modelling approach. A 108-case study varied application software development kit (SDK) tracing across absent, partial and full conditions in three public applications. Discovered dependencies were combined with declared operation requirements, deployment facts and scenario assumptions; model forecasts preceded independent execution checks. With SDK tracing disabled, the tested OpenTelemetry eBPF Instrumentation path supported all seven operations in two applications across three repetitions. Forecasts agreed exactly in 17 of 21 operation-repetition groups; the maximum state-weighted discrepancy was 4.22 percentage points. Combining observed context links supported nine additional groups under partial tracing. The results establish an acquisition path for virtual fault experiments under the tested observation constraints and identify limits imposed by protocols, evidence association and operation semantics.

## Tracing the Messy Middle: A Runtime Transaction Schema for AI Coding Agents

DevFeed: [Tracing the Messy Middle: A Runtime Transaction Schema for AI Coding Agents](<https://devfeed.tech/articles/tracing-the-messy-middle-a-runtime-transaction-schema-for-ai-coding-agents-142123.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23234>)

Author: Yuelin Hou, Haorui Li, Lu Chen, Zhen Chen

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [ai observability](<https://devfeed.tech/topics/ai-observability.md>), [tracing](<https://devfeed.tech/topics/tracing.md>), [schema design](<https://devfeed.tech/topics/schema-design.md>)

Tags: [ai-coding-agents](<https://devfeed.tech/tags/ai-coding-agents.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [atm](<https://devfeed.tech/tags/atm.md>), [causality](<https://devfeed.tech/tags/causality.md>), [coaching](<https://devfeed.tech/tags/coaching.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [fallback](<https://devfeed.tech/tags/fallback.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [local-first](<https://devfeed.tech/tags/local-first.md>), [observability](<https://devfeed.tech/tags/observability.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [productivity](<https://devfeed.tech/tags/productivity.md>), [reconciliation](<https://devfeed.tech/tags/reconciliation.md>), [retries](<https://devfeed.tech/tags/retries.md>), [routing-policy](<https://devfeed.tech/tags/routing-policy.md>), [runtime](<https://devfeed.tech/tags/runtime.md>), [schema](<https://devfeed.tech/tags/schema.md>), [sessions](<https://devfeed.tech/tags/sessions.md>), [tracing](<https://devfeed.tech/tags/tracing.md>), [transaction](<https://devfeed.tech/tags/transaction.md>)

### AI overview

The paper proposes recording AI coding agents' runtime activity as structured transactions to expose retries, fallbacks, routing choices, and abandoned sessions that final artifacts miss. Such records could support behavior-based engineering coaching, workflow improvements at failure points, and routing decisions based on total task cost rather than per-request price. The authors describe a runtime transaction schema and initial feasibility with Agent Token Manager (Atm), a local-first prototype that reconciles gateway records with transcript evidence without sending prompt content to external collectors. Scaling this approach still faces challenges: reconciling traces without shared request IDs, linking outcomes across asynchronous systems, and establishing whether interventions cause improvements.

### Source excerpt

arXiv:2611.23234v1 Announce Type: new Abstract: AI coding agents are increasingly integrated into software development, yet the field lacks a systematic feedback loop to improve their behavior in practice. Artifact datasets capture final outputs such as code patches, commits, and benchmark outcomes, but omit the messy middle of the development lifecycle: the failed retries, fallback strategies, routing decisions, and abandoned sessions that determine operational cost, reliability, and productivity. We propose an observability framework that captures these runtime behaviors as structured transaction records, enabling engineering coaching grounded in behavioral evidence, workflow optimization at the stages where tasks break down, and routing policy iteration measured by total task cost rather than per-request price. We define the runtime transaction schema this architecture requires and demonstrate initial feasibility through Agent Token Manager (Atm), a local-first prototype that reconciles gateway records and transcript evidence without exposing prompt content to external collectors. Finally, we identify three open research problems that must be solved to scale these feedback mechanisms: trace reconciliation without shared request identifiers, outcome linking across asynchronous systems, and intervention causality. Treating the linkage between sessions and artifacts as a fundamental infrastructure requirement provides a structured foundation for understanding and optimizing agent behavior.

## Why Groundcover's CEO says agents are the power users of observability

DevFeed: [Why Groundcover's CEO says agents are the power users of observability](<https://devfeed.tech/articles/why-groundcover-s-ceo-says-agents-are-the-power-users-of-observability-142011.md>)

Original publisher: [Read original article](<https://thenewstack.io/groundcover-coding-agents-observability/>)

Author: Chris J. Preimesberger

Published: 2026-11-19T03:15:24Z

Content type: article

Language: en

Sources: [Kubernetes Overview, News and Trends | The New Stack](<https://devfeed.tech/sources/kubernetes-overview-news-and-trends-the-new-stack.md>), [The New Stack](<https://devfeed.tech/sources/the-new-stack.md>)

Topics: [observability](<https://devfeed.tech/topics/observability.md>), [agent observability](<https://devfeed.tech/topics/agent-observability.md>), [tracing](<https://devfeed.tech/topics/tracing.md>), [OpenTelemetry](<https://devfeed.tech/topics/opentelemetry.md>), [eBPF](<https://devfeed.tech/topics/ebpf.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [ceo](<https://devfeed.tech/tags/ceo.md>), [groundcover](<https://devfeed.tech/tags/groundcover.md>), [kubecon-cloudnativecon-na-2026](<https://devfeed.tech/tags/kubecon-cloudnativecon-na-2026.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [observability](<https://devfeed.tech/tags/observability.md>), [podcast](<https://devfeed.tech/tags/podcast.md>), [power-users](<https://devfeed.tech/tags/power-users.md>), [sponsor-groundcover](<https://devfeed.tech/tags/sponsor-groundcover.md>), [sponsored](<https://devfeed.tech/tags/sponsored.md>), [sponsored-podcast](<https://devfeed.tech/tags/sponsored-podcast.md>), [video](<https://devfeed.tech/tags/video.md>)

### AI overview

Groundcover exposes production telemetry to coding agents through MCP, so they can use traces to investigate latency and observed behavior to guide tests; its own agents also use telemetry from staging in code reviews. The article describes agent runs as potentially lasting hours and spanning thousands of tool calls, making prompts, tool calls, and evaluations useful for understanding behavior beyond whether a run completed. Groundcover collects application and infrastructure signals with an eBPF sensor without requiring an SDK for every service, while supporting OpenTelemetry and custom instrumentation for more detail. Its bring-your-own-cloud model keeps telemetry and Groundcover's agent operations in the customer's cloud; customers pay for the infrastructure and Groundcover manages deployment. The CEO describes Kubernetes cost and resilience optimization as current work and agent-led incident response across code and infrastructure as a possible next step.

### Source excerpt

An AI coding agent can write a patch in minutes. Ask it whether that patch slows down the API endpoint The post Why Groundcover's CEO says agents are the power users of observability appeared first on The New Stack.

## What Is Agent Tracing? How to See Every Step an Agent Takes

DevFeed: [What Is Agent Tracing? How to See Every Step an Agent Takes](<https://devfeed.tech/articles/what-is-agent-tracing-how-to-see-every-step-an-agent-takes-141927.md>)

Original publisher: [Read original article](<https://montecarlo.ai/blog-agent-tracing>)

Author: Ruslan Sultanov

Published: 2026-11-18T17:20:39Z

Content type: article

Language: en

Sources: [Monte Carlo](<https://devfeed.tech/sources/monte-carlo.md>)

Topics: [ai observability](<https://devfeed.tech/topics/ai-observability.md>), [tracing](<https://devfeed.tech/topics/tracing.md>), [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [Multi-step AI processes](<https://devfeed.tech/topics/multi-step-ai-processes.md>), [Latency](<https://devfeed.tech/topics/latency.md>)

Tags: [accountability](<https://devfeed.tech/tags/accountability.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agent-spans](<https://devfeed.tech/tags/agent-spans.md>), [agent-tracing](<https://devfeed.tech/tags/agent-tracing.md>), [agent-tracing-vs-logging](<https://devfeed.tech/tags/agent-tracing-vs-logging.md>), [agent-trust](<https://devfeed.tech/tags/agent-trust.md>), [agent-trust-glossary](<https://devfeed.tech/tags/agent-trust-glossary.md>), [ai-agent-tracing](<https://devfeed.tech/tags/ai-agent-tracing.md>), [black-box](<https://devfeed.tech/tags/black-box.md>), [customer](<https://devfeed.tech/tags/customer.md>), [glossary](<https://devfeed.tech/tags/glossary.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [latency](<https://devfeed.tech/tags/latency.md>), [llm-tracing](<https://devfeed.tech/tags/llm-tracing.md>), [logs](<https://devfeed.tech/tags/logs.md>), [model](<https://devfeed.tech/tags/model.md>), [opentelemetry-genai-semantic-conventions](<https://devfeed.tech/tags/opentelemetry-genai-semantic-conventions.md>), [order](<https://devfeed.tech/tags/order.md>), [request](<https://devfeed.tech/tags/request.md>), [response](<https://devfeed.tech/tags/response.md>), [service](<https://devfeed.tech/tags/service.md>), [tools](<https://devfeed.tech/tags/tools.md>), [trace-ai-agents](<https://devfeed.tech/tags/trace-ai-agents.md>), [tracing](<https://devfeed.tech/tags/tracing.md>), [what-is](<https://devfeed.tech/tags/what-is.md>), [what-is-agent-tracing](<https://devfeed.tech/tags/what-is-agent-tracing.md>)

### AI overview

Agent tracing records an AI agent's full run as linked spans, capturing steps such as planning, retrieval, model calls, and tool use alongside inputs, outputs, timing, and metadata. This makes silent failures, retries, quality issues, and cost or latency drivers easier to investigate than ordinary logs or service-focused distributed traces. The excerpt recommends OpenTelemetry GenAI conventions, trace IDs and parent-span IDs, versioning prompts and models, linking retrieved items to source data, and scoring traces. Capture sensitive prompts carefully: redact personal data and set retention policies. OpenTelemetry conventions remain in development, so pin versions. Traces also cannot reveal upstream data freshness problems without lineage or data monitoring. Teams can store traces in their own warehouse, but need tooling to query, score, and alert on them.

### Source excerpt

An agent tells a customer their order shipped. It didn't. Every service returned 200 OK, latency was normal, and the logs show a request in and a response out. What happened in between, which documents the agent read, which tools it called, what the model was told, is a black box. Agent tracing opens that ...

## AWS observability maturity assessment: from guesswork to a blueprint

DevFeed: [AWS observability maturity assessment: from guesswork to a blueprint](<https://devfeed.tech/articles/aws-observability-maturity-assessment-from-guesswork-to-a-blueprint-137732.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/mt/aws-observability-maturity-assessment-from-guesswork-to-a-blueprint/>)

Author: Nilo Bustani

Published: 2026-11-16T15:40:28Z

Content type: article

Language: en

Sources: [AWS Cloud Operations Blog](<https://devfeed.tech/sources/aws-cloud-operations-blog.md>)

Topics: [observability](<https://devfeed.tech/topics/observability.md>), [Amazon CloudWatch](<https://devfeed.tech/topics/amazon-cloudwatch.md>), [Amazon CloudWatch Logs](<https://devfeed.tech/topics/amazon-cloudwatch-logs.md>), [OpenTelemetry](<https://devfeed.tech/topics/opentelemetry.md>), [tracing](<https://devfeed.tech/topics/tracing.md>), [Production Engineering](<https://devfeed.tech/topics/production-engineering.md>), [AWS CloudFormation](<https://devfeed.tech/topics/aws-cloudformation.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-and-machine-learning](<https://devfeed.tech/tags/ai-and-machine-learning.md>), [ai-assisted](<https://devfeed.tech/tags/ai-assisted.md>), [ai-ml](<https://devfeed.tech/tags/ai-ml.md>), [ai-powered](<https://devfeed.tech/tags/ai-powered.md>), [alarm](<https://devfeed.tech/tags/alarm.md>), [alerting](<https://devfeed.tech/tags/alerting.md>), [amazon-cloudwatch](<https://devfeed.tech/tags/amazon-cloudwatch.md>), [amazon-cloudwatch-logs](<https://devfeed.tech/tags/amazon-cloudwatch-logs.md>), [amazon-ec2](<https://devfeed.tech/tags/amazon-ec2.md>), [amazon-elastic-compute-cloud-amazon-ec2](<https://devfeed.tech/tags/amazon-elastic-compute-cloud-amazon-ec2.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-simple-storage-service-amazon-s3](<https://devfeed.tech/tags/amazon-simple-storage-service-amazon-s3.md>), [assessment](<https://devfeed.tech/tags/assessment.md>), [aws](<https://devfeed.tech/tags/aws.md>), [best-practices](<https://devfeed.tech/tags/best-practices.md>), [blueprint](<https://devfeed.tech/tags/blueprint.md>), [management-governance](<https://devfeed.tech/tags/management-governance.md>), [management-tools](<https://devfeed.tech/tags/management-tools.md>), [maturity](<https://devfeed.tech/tags/maturity.md>), [monitoring-and-observability](<https://devfeed.tech/tags/monitoring-and-observability.md>), [observability](<https://devfeed.tech/tags/observability.md>), [prioritization](<https://devfeed.tech/tags/prioritization.md>)

### AI overview

The open-source AWS Observability Assessment scores an AWS environment across logs, metrics, traces, dashboards and alerting, and organizational practices, then provides evidence-backed recommendations to prioritize improvements. Its model has Reactive, Proactive, and Autonomous stages; 52 read-only discovery checks map to 17 questions and produce category and overall scores in an interactive HTML report. It runs in AWS CodeBuild, provisioned by CloudFormation, with deployment taking about five minutes and scanning about fifteen. It supports single-account runs and multi-account scans through assumed read-only roles. One exception to its read-only API calls is the EC2 CloudWatch agent check, which uses Systems Manager ssm:SendCommand. The report is a point-in-time snapshot; the article suggests repeating scans to track changes. The sample account scored 1.1/4 overall, with logs identified as its highest-leverage starting point.

### Source excerpt

Introduction Most teams know they should have good observability. Few can answer the two questions that matter: How good is our observability posture today, and what should we fix first? Without an objective way to measure, prioritization defaults to intuition, tribal knowledge, and the most recent escalation. The service that paged someone last night gets [...]

## Daemons can now send traces to Vercel Tracing

DevFeed: [Daemons can now send traces to Vercel Tracing](<https://devfeed.tech/articles/daemons-can-now-send-traces-to-vercel-tracing-133481.md>)

Original publisher: [Read original article](<https://vercel.com/changelog/vercel-daemons-can-now-send-traces-to-vercel-tracing>)

Author: Gal Schlezinger

Published: 2026-11-13T22:24:00Z

Content type: release

Language: en

Sources: [Vercel News](<https://devfeed.tech/sources/vercel-news.md>)

Topics: [tracing](<https://devfeed.tech/topics/tracing.md>), [OpenTelemetry](<https://devfeed.tech/topics/opentelemetry.md>), [Vercel](<https://devfeed.tech/topics/vercel.md>), [observability pipelines](<https://devfeed.tech/topics/observability-pipelines.md>)

Tags: [automatic](<https://devfeed.tech/tags/automatic.md>), [cli](<https://devfeed.tech/tags/cli.md>), [export](<https://devfeed.tech/tags/export.md>), [http](<https://devfeed.tech/tags/http.md>), [json](<https://devfeed.tech/tags/json.md>), [node-js](<https://devfeed.tech/tags/node-js.md>), [opentelemetry](<https://devfeed.tech/tags/opentelemetry.md>), [sdk](<https://devfeed.tech/tags/sdk.md>), [search](<https://devfeed.tech/tags/search.md>), [service](<https://devfeed.tech/tags/service.md>), [socket](<https://devfeed.tech/tags/socket.md>), [spans](<https://devfeed.tech/tags/spans.md>), [traces](<https://devfeed.tech/tags/traces.md>), [unix](<https://devfeed.tech/tags/unix.md>), [vercel](<https://devfeed.tech/tags/vercel.md>)

### AI overview

Vercel Daemons can export OpenTelemetry spans to Vercel Tracing through a per-daemon Unix socket whose path is provided in `VERCEL_OTEL_SOCKET_PATH`. For Node.js daemons, `@vercel/otel` 2.3.0 or later automatically detects the socket with default configuration; initialize OpenTelemetry before the worker starts. The Vercel CLI can search traces by project and service name and return JSON, including daemon spans from the last hour.

### Source excerpt

Vercel Daemons can now send OpenTelemetry traces to Vercel Tracing, allowing you and your agents to inspect background worker operations alongside the rest of your project's traces. Vercel provides each daemon with a Unix socket for receiving OTLP/HTTP traces and sets the socket's path in VERCEL_OTEL_SOCKET_PATH. Upgrading @vercel/otel to version 2.3.0 or later enables automatic trace export for Node.js daemons using the default configuration. The SDK detects this socket and sends the spans your worker records to Vercel. These spans are available through the Vercel CLI, which can search by service name within a project and return JSON for your agents. Learn more in the daemon tracing documentation. Read more

## Towards Runtime-Aware Observability for Java Virtual Threads

DevFeed: [Towards Runtime-Aware Observability for Java Virtual Threads](<https://devfeed.tech/articles/towards-runtime-aware-observability-for-java-virtual-threads-131704.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.14999>)

Author: Wenjie Ding

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [observability](<https://devfeed.tech/topics/observability.md>), [tracing](<https://devfeed.tech/topics/tracing.md>), [eBPF](<https://devfeed.tech/topics/ebpf.md>), [openjdk](<https://devfeed.tech/topics/openjdk.md>)

Tags: [cs-os](<https://devfeed.tech/tags/cs-os.md>), [cs-pf](<https://devfeed.tech/tags/cs-pf.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [java](<https://devfeed.tech/tags/java.md>), [observability](<https://devfeed.tech/tags/observability.md>), [runtime](<https://devfeed.tech/tags/runtime.md>), [virtual-threads](<https://devfeed.tech/tags/virtual-threads.md>)

### AI overview

Java virtual threads can break OS-level tracing that joins requests to kernel events by thread ID, because virtual threads move between carrier threads. The proposed design binds a 64-byte off-heap trace buffer at a stable address to each virtual thread and publishes it to eBPF through four USDT probes in patched OpenJDK. In tests, it attributed all 768 markers correctly; a naive two-probe design silently misattributed context. With no consumer attached, publication costs at most about 33 ns per transition, while buffer allocation costs about 0.37 μs per virtual thread--small on transition-heavy workloads but nearly doubling a creation-only microbenchmark. The compared JVMTI channel costs about 20% on transition-heavy workloads and 1.8 μs per virtual thread in the creation-only test. For readers that cache record addresses, the abstract finds only in-place updates guarded by the valid flag sound among the two OTEP 4947 strategies discussed.

### Source excerpt

arXiv:2611.14999v1 Announce Type: cross Abstract: OS-level observability tools join application requests to kernel events through the OS thread ID, assuming that a unit of work occupies one OS thread until it completes. Java virtual threads break this assumption: many virtual threads are multiplexed onto a few carrier threads, so a stable thread ID hides a changing unit of work, and the join fails silently. Existing virtual-thread-aware channels republish context or identity on every mount and unmount, and the JVMTI-based one pays this cost for the lifetime of the JVM, whether or not an observer is attached. We bind a 64-byte off-heap trace buffer to each virtual thread, at a fixed address for its lifetime, and publish the address to eBPF consumers through four USDT probes in a patched OpenJDK; a re-implemented JVMTI channel over the same buffer serves as a controlled comparison. The four-probe design attributed all 768 test markers correctly on both the content and the identity plane, while the naive two-probe design silently misattributes trace context. With no consumer attached, publication costs at most about 33 ns per transition; the remaining dormant cost is buffer allocation, about 0.37 microseconds per virtual thread, which is negligible on transition-heavy workloads but nearly doubles the cost of a creation-only microbenchmark. The JVMTI channel costs about 20% on transition-heavy workloads and about 1.8 microseconds per virtual thread on the creation-only one. We also show that, of the two update strategies OTEP 4947 permits, only in-place rewriting guarded by the record's valid flag is sound for readers that cache the record's address, as lifecycle-event readers do.

## Best Tools for AI Agent Observability in Banking

DevFeed: [Best Tools for AI Agent Observability in Banking](<https://devfeed.tech/articles/best-tools-for-ai-agent-observability-in-banking-131241.md>)

Original publisher: [Read original article](<https://www.lyzr.ai/blog/ai-agent-observability-tools-for-banks/>)

Author: Lyzr Team

Published: 2026-11-12T10:05:55Z

Content type: comparison

Language: en

Sources: [Lyzr.ai](<https://devfeed.tech/sources/lyzr-ai.md>)

Topics: [agent observability](<https://devfeed.tech/topics/agent-observability.md>), [observability ai agents](<https://devfeed.tech/topics/observability-ai-agents.md>), [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [OpenTelemetry](<https://devfeed.tech/topics/opentelemetry.md>), [tracing](<https://devfeed.tech/topics/tracing.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [observability](<https://devfeed.tech/topics/observability.md>), [Grafana Cloud](<https://devfeed.tech/topics/grafana-cloud.md>), [human in the loop agent](<https://devfeed.tech/topics/human-in-the-loop-agent.md>), [llamaindex](<https://devfeed.tech/topics/llamaindex.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agent-observability](<https://devfeed.tech/tags/agent-observability.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [ai-agent-observability](<https://devfeed.tech/tags/ai-agent-observability.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [banking](<https://devfeed.tech/tags/banking.md>), [best](<https://devfeed.tech/tags/best.md>), [observability](<https://devfeed.tech/tags/observability.md>), [tools](<https://devfeed.tech/tags/tools.md>)

### AI overview

The comparison separates three banking needs: tracing agent runs, controlling agent actions, and preserving an examiner-ready governance record. Fiddler, Arthur, Arize Phoenix, Grafana Cloud and OpenObserve describe run observability; Opencontroller authorizes tool calls, records activity and can quarantine agents; Madison links rules, obligations, policies, controls, tests and evidence to a named approver but does not trace runs. The vendors differ in deployment and enforcement: Phoenix and OpenObserve describe self-hosted options, Grafana Cloud is described as cloud-based, Fiddler claims inline allow/block/redact, while Arthur returns a verdict for the application to enforce. The guide recommends inventorying agents, deciding what trace content may be captured, choosing deployment and enforcement models, and rehearsing a real run back to its governing rule. OpenTelemetry GenAI conventions provide a common trace structure, but remain under active development; prompt and response content is off by default.

### Source excerpt

AI agent observability in banking means being able to answer "what did the agent do, and why?" with a record that survives an examination: the...

## Guide to observability with Red Hat OpenShift 5

DevFeed: [Guide to observability with Red Hat OpenShift 5](<https://devfeed.tech/articles/guide-to-observability-with-red-hat-openshift-5-129634.md>)

Original publisher: [Read original article](<https://www.redhat.com/en/blog/guide-observability-red-hat-openshift-5>)

Author: Simon Herlofsson; Eric Evans; Vanessa Martini; Jamie Parker; Roger Florén

Published: 2026-11-11T00:00:00Z

Content type: article

Language: en

Sources: [Red Hat Blog](<https://devfeed.tech/sources/red-hat-blog.md>)

Topics: [observability](<https://devfeed.tech/topics/observability.md>), [OpenTelemetry](<https://devfeed.tech/topics/opentelemetry.md>), [AIOps for Kubernetes](<https://devfeed.tech/topics/aiops-for-kubernetes.md>), [tracing](<https://devfeed.tech/topics/tracing.md>), [Production Engineering](<https://devfeed.tech/topics/production-engineering.md>), [eBPF](<https://devfeed.tech/topics/ebpf.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [alert](<https://devfeed.tech/tags/alert.md>), [alertmanager](<https://devfeed.tech/tags/alertmanager.md>), [automatically](<https://devfeed.tech/tags/automatically.md>), [autonomous](<https://devfeed.tech/tags/autonomous.md>), [cluster](<https://devfeed.tech/tags/cluster.md>), [diagnosis](<https://devfeed.tech/tags/diagnosis.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [incident](<https://devfeed.tech/tags/incident.md>), [incident-response](<https://devfeed.tech/tags/incident-response.md>), [investigation](<https://devfeed.tech/tags/investigation.md>), [language](<https://devfeed.tech/tags/language.md>), [lightspeed](<https://devfeed.tech/tags/lightspeed.md>), [management](<https://devfeed.tech/tags/management.md>), [observability](<https://devfeed.tech/tags/observability.md>), [openshift](<https://devfeed.tech/tags/openshift.md>), [openshift-5](<https://devfeed.tech/tags/openshift-5.md>), [red-hat-openshift-5](<https://devfeed.tech/tags/red-hat-openshift-5.md>), [red-hat-openshift-lightspeed](<https://devfeed.tech/tags/red-hat-openshift-lightspeed.md>), [reliability](<https://devfeed.tech/tags/reliability.md>)

### AI overview

The excerpt describes Red Hat OpenShift 5 observability across troubleshooting, dashboards, monitoring, logging, and tracing. With the Alertmanager adapter configured, Lightspeed's autonomous troubleshooting preview investigates alerts and stages a remediation plan for administrator approval; failed fixes escalate to an operator. Its assistant can also summarize metrics, events, logs, and traces, while AI-generated Perses panels expose their PromQL, time ranges, and metrics for review. Cluster Observability Operator 1.6 adds default Thanos data sources, console-based project creation, and bundled plug-ins for air-gapped use. Monitoring guidance covers choosing a supported Prometheus remote-write version, the newly enabled NVMe collector and its storage alerts, and the ClusterMonitoring custom resource's preview status and precedence rules. Logging 6.7 adds Loki TSDB indexing, backlog filtering, Elasticsearch failover, AWS PrivateLink, and disk buffering. OpenTelemetry and tracing updates include webhook and Tempo high availability, preview eBPF instrumentation, telemetry processing components, and trace access through MCP and Lightspeed audit runs.

### Source excerpt

In Red Hat OpenShift 5, observability, powered by Red Hat OpenShift Lightspeed, transforms cluster management through 3 core capabilities designed to accelerate incident response while maintaining enterprise-grade safety and human control.Autonomous troubleshootingAfter you configure the Alertmanager adapter, Red Hat OpenShift Lightspeed takes it from there: The moment an alert fires, it automatically launches an investigation, functioning like a virtual site reliability engineer (SRE) without requiring a new alerting language. The agent inspects the cluster state, drafts an initial diagnosis,

## WasmLens: Automatic Glass-Box Instrumentation for Composed WebAssembly Applications

DevFeed: [WasmLens: Automatic Glass-Box Instrumentation for Composed WebAssembly Applications](<https://devfeed.tech/articles/wasmlens-automatic-glass-box-instrumentation-for-composed-webassembly-applications-127272.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.12710>)

Author: Michiel Van Kenhove, Ewout Verlinde, Jos\\'e Santos, Filip De Turck, Bruno Volckaert, Merlijn Sebrechts

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Wasm components](<https://devfeed.tech/topics/wasm-components.md>), [tracing](<https://devfeed.tech/topics/tracing.md>), [WebAssembly](<https://devfeed.tech/topics/web-assembly.md>), [middleware](<https://devfeed.tech/topics/middleware.md>)

Tags: [binaries](<https://devfeed.tech/tags/binaries.md>), [boundary](<https://devfeed.tech/tags/boundary.md>), [composition](<https://devfeed.tech/tags/composition.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [graph](<https://devfeed.tech/tags/graph.md>), [instrumentation](<https://devfeed.tech/tags/instrumentation.md>), [latency](<https://devfeed.tech/tags/latency.md>), [middleware](<https://devfeed.tech/tags/middleware.md>), [observability](<https://devfeed.tech/tags/observability.md>), [overhead](<https://devfeed.tech/tags/overhead.md>), [runtime](<https://devfeed.tech/tags/runtime.md>), [traces](<https://devfeed.tech/tags/traces.md>), [tracing](<https://devfeed.tech/tags/tracing.md>), [webassembly](<https://devfeed.tech/tags/webassembly.md>)

### AI overview

WasmLens reconstructs end-to-end traces across black-box WebAssembly components by injecting middleware into composed binaries, using instrumentation points derived from the composition graph. Its per-component span-stack tracing design restores parent-child relationships across component boundaries without changing component interfaces or source code; the abstract says this design was adopted upstream. In the reported measurements, mean request latency rose from 238 μs to 454 μs across six boundaries. Increasing the expected crossings from six to eleven raised latency to 514 μs, adding 12 μs per crossing, of which 10.4 μs was attributed to instrumentation. The authors characterize this overhead as additive with composition depth.

### Source excerpt

arXiv:2611.12710v1 Announce Type: new Abstract: The WebAssembly Component Model enables language-independent composition of independently developed components, but complicates observability. Existing tracing support flattens execution hierarchy by attributing every span created inside a composition to a single host-level parent context, so execution across component boundaries cannot be reconstructed. The problem is amplified because components are typically deployed as black boxes whose internals cannot be modified or inspected by the system integrator. To address these limitations, we propose a revised wasi:otel/tracing design in which the runtime maintains a per-component span stack, restoring parent-child relationships across component boundaries without changing component interfaces or source code. This change has since been adopted upstream. Building on this design, we present WasmLens, an automatic instrumentation tool that derives instrumentation points from a composition graph and injects tracing middleware into already composed binaries, yielding end-to-end traces for black-box compositions with no developer configuration. This automatic instrumentation raises mean request latency from $238$ $\mu$s to $454$ $\mu$s across six component boundaries. Raising the expected number of boundary crossings from 6 to 11 further increases latency to $514$ $\mu$s, an incremental cost of $12$ $\mu$s per crossing, of which $10.4$ $\mu$s is attributable to instrumentation. These results show that instrumentation overhead is additive in composition depth rather than proportional to baseline execution time.

## The observability UI as we know it is dead. So what comes next?

DevFeed: [The observability UI as we know it is dead. So what comes next?](<https://devfeed.tech/articles/the-observability-ui-as-we-know-it-is-dead-so-what-comes-next-124830.md>)

Original publisher: [Read original article](<https://clickhouse.com/blog/observability-ui-is-dead-whats-next>)

Author: Alex Fedotyev

Published: 2026-11-09T17:00:00Z

Content type: article

Language: en

Sources: [ClickHouse Blog](<https://devfeed.tech/sources/clickhouse-blog.md>)

Topics: [observability](<https://devfeed.tech/topics/observability.md>), [AI for interactive applications](<https://devfeed.tech/topics/ai-for-interactive-applications.md>), [User interface design](<https://devfeed.tech/topics/ui-design.md>), [Data visualization](<https://devfeed.tech/topics/data-visualization.md>), [OpenTelemetry](<https://devfeed.tech/topics/opentelemetry.md>), [tracing](<https://devfeed.tech/topics/tracing.md>)

Tags: [about](<https://devfeed.tech/tags/about.md>), [accessible](<https://devfeed.tech/tags/accessible.md>), [actions](<https://devfeed.tech/tags/actions.md>), [adversarial](<https://devfeed.tech/tags/adversarial.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [apps](<https://devfeed.tech/tags/apps.md>), [can](<https://devfeed.tech/tags/can.md>), [clickhouse](<https://devfeed.tech/tags/clickhouse.md>), [custom](<https://devfeed.tech/tags/custom.md>), [explore](<https://devfeed.tech/tags/explore.md>), [investigation](<https://devfeed.tech/tags/investigation.md>), [need](<https://devfeed.tech/tags/need.md>), [observability](<https://devfeed.tech/tags/observability.md>), [possible](<https://devfeed.tech/tags/possible.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [questions](<https://devfeed.tech/tags/questions.md>), [right](<https://devfeed.tech/tags/right.md>), [sql](<https://devfeed.tech/tags/sql.md>), [ui](<https://devfeed.tech/tags/ui.md>)

### AI overview

ClickHouse Observability Apps lets teams describe an observability workflow in natural language and have an agent build an interactive app over live data, with connected views, visualizations, and reusable investigation tools. SQL enables analyses beyond fixed endpoints, including joins and business-specific logic; queries can be inspected and are tested before use. Apps reuse ClickStack APIs and UI components, while OpenTelemetry conventions and team schema context help the agent interpret telemetry. Generated apps run in a sandboxed iframe and can request only validated actions; queries are read-only and use the viewing user's database permissions, so sharing an app does not share its creator's access. The product is in private preview. The authors say a successful query may still answer the wrong question, and they are improving evaluations, interaction testing, linting, and adversarial review. They also say prompt-injection robustness is not yet solved.

### Source excerpt

Your observability UI shouldn't limit the questions you can ask. Explore how ClickHouse Observability Apps turns a prompt into custom investigation experiences, why SQL makes it possible, and what we still need to get right.

## Vercel Tracing is now generally available

DevFeed: [Vercel Tracing is now generally available](<https://devfeed.tech/articles/vercel-tracing-is-now-generally-available-121602.md>)

Original publisher: [Read original article](<https://vercel.com/changelog/vercel-tracing-is-now-generally-available>)

Author: Luka Hartwig

Published: 2026-11-06T17:00:00Z

Content type: release

Language: en

Sources: [Vercel News](<https://devfeed.tech/sources/vercel-news.md>)

Topics: [tracing](<https://devfeed.tech/topics/tracing.md>), [Vercel](<https://devfeed.tech/topics/vercel.md>)

Tags: [automatic](<https://devfeed.tech/tags/automatic.md>), [beta](<https://devfeed.tech/tags/beta.md>), [build](<https://devfeed.tech/tags/build.md>), [cli](<https://devfeed.tech/tags/cli.md>), [dashboard](<https://devfeed.tech/tags/dashboard.md>), [database](<https://devfeed.tech/tags/database.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [flag](<https://devfeed.tech/tags/flag.md>), [hobby](<https://devfeed.tech/tags/hobby.md>), [instrumentation](<https://devfeed.tech/tags/instrumentation.md>), [kibana](<https://devfeed.tech/tags/kibana.md>), [next-js](<https://devfeed.tech/tags/next-js.md>), [node-js](<https://devfeed.tech/tags/node-js.md>), [pro](<https://devfeed.tech/tags/pro.md>), [retention](<https://devfeed.tech/tags/retention.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [search](<https://devfeed.tech/tags/search.md>), [spans](<https://devfeed.tech/tags/spans.md>), [tracing](<https://devfeed.tech/tags/tracing.md>), [vercel](<https://devfeed.tech/tags/vercel.md>)

### AI overview

Vercel Tracing is generally available on all plans, with pricing unchanged from beta. The dashboard's Traces page brings project or team traces and spans together, supports filtering, KQL search, and live updates; the CLI uses the same search syntax and can inspect individual traces or manage sampling rules. Starting with Next.js 16.5, new Vercel projects with at least one Node.js function enable tracing after the first successful build, with 100% request sampling in all environments and build-time instrumentation that adds framework spans. Remove all sampling rules in Settings -> Tracing to opt out. Trace retention is now 3 days on Hobby, 14 on Pro, and 14 on Enterprise.

### Source excerpt

Vercel Tracing is now generally available on all plans. You can now search and inspect traces from the dashboard or the Vercel CLI, and new Next.js projects get tracing by default. Pricing is unchanged from the beta. Browse and search traces in the dashboard In beta, you could only open a trace per request from the Logs page. The new Traces page shows every trace and span in one place, for a project or your whole team. Switch between a Spans list and a Traces list, filter each, and search with Kibana Query Language (KQL). A Live mode streams new traces as they arrive. A query like span.name:db* AND span.duration > 500 narrows to slow database spans by name and duration. Search traces from the CLI The Traces page uses the same KQL as vercel traces search --query, so a query built in the dashboard runs in the CLI unchanged. Search covers every project in your team, or one project with the --project flag. Inspect a single trace with vercel traces inspect <request-id>, and manage a project's sampling rules with vercel traces config. Automatic tracing for new Next.js projects In beta, nothing was collected until you added a sampling rule. Starting with Next.js 16.5, new projects created on Vercel that include at least one Node.js function get tracing turned on after their first successful build. Vercel adds one sampling rule tracing 100% of requests in all environments and injects instrumentation at build time, so traces include Next.js framework spans with no setup. To opt out, remove all sampling rules in Settings -> Tracing. Longer retention In beta, traces followed your runtime log retention. Hobby now keeps traces for 3 days instead of 1 hour, Pro for 14 days instead of 1 day, and Enterprise for 14 days instead of 3 days. Read the Tracing documentation to learn more. Read more

## Manage Kubernetes observability with DatadogInstrumentation

DevFeed: [Manage Kubernetes observability with DatadogInstrumentation](<https://devfeed.tech/articles/manage-kubernetes-observability-with-datadoginstrumentation-121095.md>)

Original publisher: [Read original article](<https://www.datadoghq.com/blog/manage-kubernetes-observability-with-datadoginstrumentation/>)

Author: Mathew Estafanous; Wassim Dhif; David Lentz

Published: 2026-11-05T00:00:00Z

Content type: tutorial

Language: en

Sources: [Datadog](<https://devfeed.tech/sources/datadog-the-monitor-blog.md>)

Topics: [observability](<https://devfeed.tech/topics/observability.md>), [tracing](<https://devfeed.tech/topics/tracing.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [apm](<https://devfeed.tech/tags/apm.md>), [collection](<https://devfeed.tech/tags/collection.md>), [configuration-management](<https://devfeed.tech/tags/configuration-management.md>), [configure](<https://devfeed.tech/tags/configure.md>), [containers](<https://devfeed.tech/tags/containers.md>), [datadog-agent](<https://devfeed.tech/tags/datadog-agent.md>), [gitops](<https://devfeed.tech/tags/gitops.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [http](<https://devfeed.tech/tags/http.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [log-management](<https://devfeed.tech/tags/log-management.md>), [logs](<https://devfeed.tech/tags/logs.md>), [monitoring-as-code](<https://devfeed.tech/tags/monitoring-as-code.md>), [observability](<https://devfeed.tech/tags/observability.md>), [profiling](<https://devfeed.tech/tags/profiling.md>), [tracing](<https://devfeed.tech/tags/tracing.md>)

### AI overview

DatadogInstrumentation lets Kubernetes workload owners manage checks, logs, and APM settings in one resource while platform teams retain control of Agent deployments. The Cluster Agent validates configuration at apply time and reports readiness conditions. In the checkout example, an HTTP check, log settings, and Java tracing are moved out of pod annotations and Agent targeting rules; check and log updates take effect without an application rollout or Agent restart. The team later adds OpenMetrics metrics and email masking the same way. Tracing changes, including enabling profiling through the Java tracer, apply to newly created pods and therefore require a rollout to replace existing pods. The described setup requires Agent and Cluster Agent v7.85 or later, plus Operator v1.31 or later or Helm chart v3.255.0 or later.

### Source excerpt

Configure Kubernetes checks, logs, tracing, and profiling with DatadogInstrumentation while keeping workload changes separate from Agent deployments.

## GHOST Driver

DevFeed: [GHOST Driver](<https://devfeed.tech/articles/ghost-driver-116476.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.03649>)

Author: Azhar Shaikh

Published: 2026-11-04T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [eBPF](<https://devfeed.tech/topics/ebpf.md>), [gpu monitoring](<https://devfeed.tech/topics/gpu-monitoring.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [dma](<https://devfeed.tech/topics/dma.md>), [tracing](<https://devfeed.tech/topics/tracing.md>), [OpenTelemetry](<https://devfeed.tech/topics/opentelemetry.md>), [Prometheus](<https://devfeed.tech/topics/prometheus.md>), [Linux](<https://devfeed.tech/topics/linux.md>), [dmaengine](<https://devfeed.tech/topics/dmaengine.md>)

Tags: [ai-inference](<https://devfeed.tech/tags/ai-inference.md>), [attribution](<https://devfeed.tech/tags/attribution.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-os](<https://devfeed.tech/tags/cs-os.md>), [dma](<https://devfeed.tech/tags/dma.md>), [driver](<https://devfeed.tech/tags/driver.md>), [drm](<https://devfeed.tech/tags/drm.md>), [ebpf](<https://devfeed.tech/tags/ebpf.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpu-clusters](<https://devfeed.tech/tags/gpu-clusters.md>), [gpu-utilization](<https://devfeed.tech/tags/gpu-utilization.md>), [interrupt](<https://devfeed.tech/tags/interrupt.md>), [latency](<https://devfeed.tech/tags/latency.md>), [linux-kernel](<https://devfeed.tech/tags/linux-kernel.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [mig](<https://devfeed.tech/tags/mig.md>), [opentelemetry](<https://devfeed.tech/tags/opentelemetry.md>), [overhead](<https://devfeed.tech/tags/overhead.md>), [profiling](<https://devfeed.tech/tags/profiling.md>), [prometheus](<https://devfeed.tech/tags/prometheus.md>), [vgpu](<https://devfeed.tech/tags/vgpu.md>)

### AI overview

GHOST-Driver aims to explain why high GPU utilization readings can coexist with lower effective throughput by tracing delays inside the Linux kernel driver, which userspace tools such as CUPTI do not expose. The eBPF framework follows six submission stages, from ioctl entry and GEM buffer validation through fence handling, command emission, DMA scheduling, and interrupt signaling. It correlates traces with tenant vGPU context IDs for attribution in SR-IOV and MIG environments and exports metrics to Prometheus and OpenTelemetry. The abstract reports sub-microsecond resolution and CPU overhead below 2% in production.

### Source excerpt

arXiv:2611.03649v1 Announce Type: cross Abstract: AI inference workloads running on GPU clusters exhibit a well-documented utilization paradox: monitoring tools such as nvidia-smi regularly report GPU utilization figures above 80%, yet effective compute throughput is substantially lower. A significant fraction of reported utilization time is consumed by fence waits, driver submission overhead, DMA scheduling latency, and interrupt coalescing delays, none of which is visible to userspace profiling tools. These hidden latency components are embedded deep inside the Linux kernel GPU driver stack, primarily within the Direct Rendering Manager (DRM) subsystem, GEM buffer object lifecycle, and dma_fence synchronization infrastructure. Existing vendor profiling tools like CUPTI operate at the CUDA userspace boundary and provide no visibility into kernel-side driver paths. We present GHOST-Driver (GPU Host-Side Observability and Tracing for Drivers), a non-intrusive eBPF-based framework that attaches to stable DRM kernel tracepoints and select kprobes to expose the full latency decomposition of the GPU driver submission path with sub-microsecond resolution and CPU overhead below 2% in production. GHOST-Driver tracks six key submission stages (ioctl entry, GEM BO validation, dma_fence dependency resolution, command ring emission, DMA engine scheduling, and interrupt fence signaling) and correlates them with per-tenant vGPU context IDs for multi-tenant attribution in SR-IOV and MIG environments, exporting trace metrics to Prometheus and OpenTelemetry.

## Compare AI agent changes with Online Experiments in Agent Observability

DevFeed: [Compare AI agent changes with Online Experiments in Agent Observability](<https://devfeed.tech/articles/compare-ai-agent-changes-with-online-experiments-in-agent-observability-118927.md>)

Original publisher: [Read original article](<https://www.datadoghq.com/blog/online-experiments-agent-observability/>)

Author: Alex Barksdale; Yakov Shapiro; Demetri Pananos; Miguel Tulla Lizardi

Published: 2026-11-04T00:00:00Z

Content type: article

Language: en

Sources: [Datadog](<https://devfeed.tech/sources/datadog-the-monitor-blog.md>)

Topics: [experiments](<https://devfeed.tech/topics/experiments.md>), [agent observability](<https://devfeed.tech/topics/agent-observability.md>), [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [product-experimentation](<https://devfeed.tech/topics/product-experimentation.md>), [observability](<https://devfeed.tech/topics/observability.md>), [tracing](<https://devfeed.tech/topics/tracing.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agent-observability](<https://devfeed.tech/tags/agent-observability.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [can](<https://devfeed.tech/tags/can.md>), [changes](<https://devfeed.tech/tags/changes.md>), [compare](<https://devfeed.tech/tags/compare.md>), [cost](<https://devfeed.tech/tags/cost.md>), [datadog](<https://devfeed.tech/tags/datadog.md>), [errors](<https://devfeed.tech/tags/errors.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [latency](<https://devfeed.tech/tags/latency.md>), [learn](<https://devfeed.tech/tags/learn.md>), [observability](<https://devfeed.tech/tags/observability.md>), [online](<https://devfeed.tech/tags/online.md>), [production](<https://devfeed.tech/tags/production.md>), [quality](<https://devfeed.tech/tags/quality.md>), [test](<https://devfeed.tech/tags/test.md>), [testing](<https://devfeed.tech/tags/testing.md>), [traffic](<https://devfeed.tech/tags/traffic.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

Datadog Online Experiments compares AI agent variants on real production traffic, pairing evaluation scores with operational measures such as latency, cost, token use, and errors. Unlike offline experiments built on predefined datasets, online evaluations can surface unanticipated requests and behaviors; random assignment against a control helps compare variants under the same period of usage. Feature Flags can adjust traffic exposure without a new deployment, and traces let teams inspect tool calls and responses behind metric changes. In Datadog's internal comparison, GPT-6 Luna's estimated cost was 44.3% lower than GPT-5.6 Luna, while mean quality scores were close (3.86 vs. 3.84); those averages did not establish equivalent quality. Duration and workflow-error estimates were higher but uncertain, and trace review found more invalid tool parameters in GPT-6 Luna, usually corrected on retry. The team addressed that issue before proceeding and identified failures as candidates for offline regression tests. The article presents online experiments as complementary to offline testing, with results also informing future evaluation datasets.

### Source excerpt

Learn how Datadog Online Experiments, available through Agent Observability, enables you to test AI agent changes against real production traffic so you can compare response quality, cost, latency, and errors.

## Claude Platform now supports native OpenTelemetry observability in Databricks MLflow

DevFeed: [Claude Platform now supports native OpenTelemetry observability in Databricks MLflow](<https://devfeed.tech/articles/claude-platform-now-supports-native-opentelemetry-observability-in-databricks-mlflow-115578.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/claude-platform-now-supports-native-opentelemetry-observability-databricks-mlflow>)

Author: Yuki Watanabe

Published: 2026-11-03T21:00:00Z

Content type: release

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [OpenTelemetry](<https://devfeed.tech/topics/opentelemetry.md>), [agent observability](<https://devfeed.tech/topics/agent-observability.md>), [tracing](<https://devfeed.tech/topics/tracing.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [observability](<https://devfeed.tech/topics/observability.md>), [databricks](<https://devfeed.tech/topics/databricks.md>), [Claude](<https://devfeed.tech/topics/claude.md>), [Claude Code](<https://devfeed.tech/topics/claude-code.md>)

Tags: [about](<https://devfeed.tech/tags/about.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agent-evaluation](<https://devfeed.tech/tags/agent-evaluation.md>), [agent-sdk](<https://devfeed.tech/tags/agent-sdk.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agents](<https://devfeed.tech/tags/agents.md>), [also](<https://devfeed.tech/tags/also.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [can](<https://devfeed.tech/tags/can.md>), [claude](<https://devfeed.tech/tags/claude.md>), [claude-platform](<https://devfeed.tech/tags/claude-platform.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [important](<https://devfeed.tech/tags/important.md>), [mlflow](<https://devfeed.tech/tags/mlflow.md>), [native](<https://devfeed.tech/tags/native.md>), [opentelemetry](<https://devfeed.tech/tags/opentelemetry.md>), [platform](<https://devfeed.tech/tags/platform.md>), [plausible](<https://devfeed.tech/tags/plausible.md>), [reading](<https://devfeed.tech/tags/reading.md>), [return](<https://devfeed.tech/tags/return.md>), [source](<https://devfeed.tech/tags/source.md>)

### AI overview

Claude Platform can export native OpenTelemetry traces from Anthropic SDKs, the Messages API, and Managed Agents to Databricks MLflow. Claude instrumentation creates spans for steps such as model requests and tool calls; Databricks ingests them through Zerobus and stores them in Unity Catalog tables linked to an MLflow experiment. This gives teams a way to inspect traces alongside other instrumented services, query governed trace data with SQL, and use MLflow tools to investigate failures, review selected cases, detect recurring issues, and evaluate agent changes against shared cases. Setup includes creating an experiment with Unity Catalog trace storage and configuring the Databricks destination; detailed permission and SQL warehouse requirements are in the setup guide. Captured inputs and outputs are available only when content mode is set to "full." MLflow Issues is described as Beta.

### Source excerpt

An agent can return a plausible answer while missing an important finding or reading the same source repeatedly...

## OpenTelemetry eBPF Instrumentation reaches v1.0.0

DevFeed: [OpenTelemetry eBPF Instrumentation reaches v1.0.0](<https://devfeed.tech/articles/opentelemetry-ebpf-instrumentation-reaches-v1-0-0-109296.md>)

Original publisher: [Read original article](<https://opentelemetry.io/blog/2026/obi-v1-release/>)

Author: OpenTelemetry Authors; Docs CC BY

Published: 2026-10-30T15:42:24Z

Content type: release

Language: en

Sources: [OpenTelemetry](<https://devfeed.tech/sources/blog-on-opentelemetry.md>)

Topics: [OpenTelemetry](<https://devfeed.tech/topics/opentelemetry.md>), [eBPF](<https://devfeed.tech/topics/ebpf.md>), [telemetry](<https://devfeed.tech/topics/telemetry.md>), [tracing](<https://devfeed.tech/topics/tracing.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-applications](<https://devfeed.tech/tags/ai-applications.md>), [alerts](<https://devfeed.tech/tags/alerts.md>), [announce](<https://devfeed.tech/tags/announce.md>), [applications](<https://devfeed.tech/tags/applications.md>), [compatibility](<https://devfeed.tech/tags/compatibility.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [dependencies](<https://devfeed.tech/tags/dependencies.md>), [ebpf](<https://devfeed.tech/tags/ebpf.md>), [instrumentation](<https://devfeed.tech/tags/instrumentation.md>), [linux](<https://devfeed.tech/tags/linux.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [migration](<https://devfeed.tech/tags/migration.md>), [observability](<https://devfeed.tech/tags/observability.md>), [opentelemetry](<https://devfeed.tech/tags/opentelemetry.md>), [release](<https://devfeed.tech/tags/release.md>), [telemetry](<https://devfeed.tech/tags/telemetry.md>), [traces](<https://devfeed.tech/tags/traces.md>), [v1](<https://devfeed.tech/tags/v1.md>)

### AI overview

OBI 1.0 provides a stable foundation for collecting OpenTelemetry metrics and traces from running Linux applications without source changes, using eBPF and runtime-specific instrumentation. It documents compatibility commitments for Config v2, deployment interfaces, and supported environments, plus a telemetry migration path using Weaver-compatible schemas, release notes, and guidance. Config v1 remains supported for backward compatibility but is frozen; existing users are advised to migrate, review environment-based configuration, and compare telemetry in a canary before broad rollout. The release expands protocol, framework, runtime, and AI instrumentation, though coverage and capabilities depend on documented support and configuration. Runtime metrics remain experimental, and detailed AI payload capture requires explicit configuration. OBI can complement language agents and manual instrumentation when application-specific context is needed.

### Source excerpt

We're excited to announce v1.0.0 of OpenTelemetry eBPF Instrumentation--OBI! OBI brings OpenTelemetry observability to running Linux applications without requiring changes to their source code. It uses eBPF, together with runtime-specific instrumentation, to capture application metrics and traces across services, languages, and frameworks. You can start with request rates, errors, latency, and service dependencies, then build deeper visibility as your needs grow. This release fulfills the flagship goal we set for 2026: establishing a stable foundation for OBI. It represents a year of expanding instrumentation coverage, improving correctness, strengthening release tooling, and building the compatibility and telemetry migration practices that users need to operate OBI over time.

## AI Agent Observability vs Traditional APM: What Changes

DevFeed: [AI Agent Observability vs Traditional APM: What Changes](<https://devfeed.tech/articles/ai-agent-observability-vs-traditional-apm-what-changes-106157.md>)

Original publisher: [Read original article](<https://www.lyzr.ai/blog/ai-agent-observability-vs-traditional-apm/>)

Author: Lyzr Team

Published: 2026-10-29T07:36:11Z

Content type: article

Language: en

Sources: [Lyzr.ai](<https://devfeed.tech/sources/lyzr-ai.md>)

Topics: [agent observability](<https://devfeed.tech/topics/agent-observability.md>), [observability ai agents](<https://devfeed.tech/topics/observability-ai-agents.md>), [tracing](<https://devfeed.tech/topics/tracing.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [observability](<https://devfeed.tech/topics/observability.md>), [OpenTelemetry](<https://devfeed.tech/topics/opentelemetry.md>), [Multi Agent Systems](<https://devfeed.tech/topics/multi-agent-systems.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agent-observability](<https://devfeed.tech/tags/agent-observability.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [ai-agent-observability](<https://devfeed.tech/tags/ai-agent-observability.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [apm](<https://devfeed.tech/tags/apm.md>), [observability](<https://devfeed.tech/tags/observability.md>), [vs](<https://devfeed.tech/tags/vs.md>)

### AI overview

Traditional APM can show a healthy request even when an AI agent makes a harmful decision: a 200 response, normal latency and no exception do not establish correctness. The article recommends keeping APM for infrastructure and adding agent-run traces for model calls, tool use, retrieval and handoffs, joined to service traces with shared trace context. Because agent traces may contain customer data, set capture and redaction rules; define correctness with policy checks, evals and sampled human review; and calibrate automated judges. Adjust sampling and alerts to catch low eval scores, high per-run cost, unusual tool-call counts, guardrail triggers and new prompt or model versions, then turn failures into regression tests. OpenTelemetry GenAI spans are identified as an instrumentation option, but their status is marked "Development," so the article advises pinning versions. Observability records and evaluates behavior; it cannot itself block a call, and uninstrumented agents remain outside its view.

### Source excerpt

TL;DR You run platform engineering at a fintech, and your APM is good: golden-signal dashboards, clear SLOs, a pager rotation that works. Then on Monday,...

## Trace voice agents with Datadog Agent Observability

DevFeed: [Trace voice agents with Datadog Agent Observability](<https://devfeed.tech/articles/trace-voice-agents-with-datadog-agent-observability-107135.md>)

Original publisher: [Read original article](<https://www.datadoghq.com/blog/agent-observability-voice-agent/>)

Author: Zach Groves; Will Potts

Published: 2026-10-29T00:00:00Z

Content type: tutorial

Language: en

Sources: [Datadog](<https://devfeed.tech/sources/datadog-the-monitor-blog.md>)

Topics: [agent observability](<https://devfeed.tech/topics/agent-observability.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [tracing](<https://devfeed.tech/topics/tracing.md>), [observability](<https://devfeed.tech/topics/observability.md>), [datadog](<https://devfeed.tech/topics/datadog.md>), [OpenTelemetry](<https://devfeed.tech/topics/opentelemetry.md>)

Tags: [agent-observability](<https://devfeed.tech/tags/agent-observability.md>), [ai](<https://devfeed.tech/tags/ai.md>), [amazon-nova](<https://devfeed.tech/tags/amazon-nova.md>), [cost](<https://devfeed.tech/tags/cost.md>), [datadog](<https://devfeed.tech/tags/datadog.md>), [latency](<https://devfeed.tech/tags/latency.md>), [observability](<https://devfeed.tech/tags/observability.md>), [openai](<https://devfeed.tech/tags/openai.md>), [realtime](<https://devfeed.tech/tags/realtime.md>), [sessions](<https://devfeed.tech/tags/sessions.md>), [trace](<https://devfeed.tech/tags/trace.md>), [transcripts](<https://devfeed.tech/tags/transcripts.md>), [voice](<https://devfeed.tech/tags/voice.md>)

### AI overview

For supported voice agents, Datadog Agent Observability puts audio, transcripts, token usage, and tool calls into traces so teams can investigate what a user actually experienced. For OpenAI Realtime, it captures per-turn input and output audio and converts raw PCM16 and G.711 to playable WAV. Teams can use traces to inspect pauses, interruptions, incomplete transcripts, and tool activity; the `ml_obs.trace.time_to_first_agent_audio` metric and "Average time to first agent audio" facet help find slow turns. OpenAI Realtime token usage can also be reviewed alongside latency and traces. The excerpt says Python apps using `ddtrace` 4.13.0 or later support automatic capture; Amazon Nova 2 Sonic tracing is available in Python and Node.js tracers. ElevenLabs Realtime support is described as planned for `ddtrace` 4.17.0. OpenTelemetry apps can send supported audio message parts when spans follow GenAI semantic conventions version 1.37 or later.

### Source excerpt

Use Agent Observability to replay voice interactions and investigate latency, cost, interruptions, and transcripts across OpenAI Realtime sessions.

## Five things you can do with Sentry's Agent Tracing

DevFeed: [Five things you can do with Sentry's Agent Tracing](<https://devfeed.tech/articles/five-things-you-can-do-with-sentry-s-agent-tracing-105671.md>)

Original publisher: [Read original article](<https://blog.sentry.io/agent-tracing-use-cases/>)

Author: Anton Bjorkman

Published: 2026-10-28T09:00:00Z

Content type: tutorial

Language: en

Sources: [Sentry](<https://devfeed.tech/sources/sentry-blog.md>)

Topics: [agent observability](<https://devfeed.tech/topics/agent-observability.md>), [tracing](<https://devfeed.tech/topics/tracing.md>), [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [Multi Agent Systems](<https://devfeed.tech/topics/multi-agent-systems.md>)

Tags: [about](<https://devfeed.tech/tags/about.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agent-costs](<https://devfeed.tech/tags/agent-costs.md>), [agent-tracing](<https://devfeed.tech/tags/agent-tracing.md>), [agents](<https://devfeed.tech/tags/agents.md>), [alert](<https://devfeed.tech/tags/alert.md>), [automate](<https://devfeed.tech/tags/automate.md>), [can](<https://devfeed.tech/tags/can.md>), [conversations](<https://devfeed.tech/tags/conversations.md>), [costs](<https://devfeed.tech/tags/costs.md>), [evals](<https://devfeed.tech/tags/evals.md>), [llm](<https://devfeed.tech/tags/llm.md>), [otel](<https://devfeed.tech/tags/otel.md>), [send](<https://devfeed.tech/tags/send.md>), [sentry](<https://devfeed.tech/tags/sentry.md>), [spans](<https://devfeed.tech/tags/spans.md>), [things](<https://devfeed.tech/tags/things.md>), [tracing](<https://devfeed.tech/tags/tracing.md>), [triage](<https://devfeed.tech/tags/triage.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

Sentry Agent Tracing links prompts, model calls, tool executions, handoffs, and app spans in traces. The excerpt presents five uses: export production conversations as eval records with inputs, outputs, cost, tokens, and tool errors for scoring in Braintrust, Langfuse, promptfoo, or Arize Phoenix; use a daily Claude scheduled task with Sentry's MCP to review sampled successes and failures and file missing Linear tickets, or run a similar prompt via cron or GitHub Actions; trace Vercel AI SDK calls in a Next.js app to inspect estimated per-model and per-conversation costs and errors; use a CLI script to create an eight-widget spend dashboard and hourly threshold and anomaly alerts, with estimated cost derived from token counts in standard gen_ai spans; and add Sentry as a destination in an existing OpenTelemetry pipeline, keeping the other destination. The OTel example uses Next.js and @vercel/otel, though the text says the approach also works in other environments. Recipes assume an agent sends traces to Sentry; costs are described as estimates.

### Source excerpt

Five Sentry Agent Tracing recipes: turn agent conversations into evals, automate triage, track LLM costs, alert on spend, and send OTel spans to Sentry.

## OpenTelemetry-powered distributed tracing and Kubernetes events in OpenChoreo

DevFeed: [OpenTelemetry-powered distributed tracing and Kubernetes events in OpenChoreo](<https://devfeed.tech/articles/opentelemetry-powered-distributed-tracing-and-kubernetes-events-in-openchoreo-109909.md>)

Original publisher: [Read original article](<https://www.cncf.io/blog/2026/10/27/opentelemetry-powered-distributed-tracing-and-kubernetes-events-in-openchoreo/>)

Author: Nilushan Costa and Akila Induranga WSO2

Published: 2026-10-27T11:00:00Z

Content type: article

Language: en

Sources: [Cloud Native Computing Foundation](<https://devfeed.tech/sources/cloud-native-computing-foundation.md>)

Topics: [OpenTelemetry](<https://devfeed.tech/topics/opentelemetry.md>), [tracing](<https://devfeed.tech/topics/tracing.md>), [internal developer platform](<https://devfeed.tech/topics/internal-developer-platform.md>)

Tags: [adapter](<https://devfeed.tech/tags/adapter.md>), [adapters](<https://devfeed.tech/tags/adapters.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [annotations](<https://devfeed.tech/tags/annotations.md>), [api](<https://devfeed.tech/tags/api.md>), [api-server](<https://devfeed.tech/tags/api-server.md>), [apis](<https://devfeed.tech/tags/apis.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [auth](<https://devfeed.tech/tags/auth.md>), [aws](<https://devfeed.tech/tags/aws.md>), [azure](<https://devfeed.tech/tags/azure.md>), [backend](<https://devfeed.tech/tags/backend.md>), [blog](<https://devfeed.tech/tags/blog.md>), [developer-platforms](<https://devfeed.tech/tags/developer-platforms.md>), [distributed-tracing](<https://devfeed.tech/tags/distributed-tracing.md>), [events](<https://devfeed.tech/tags/events.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [opentelemetry](<https://devfeed.tech/tags/opentelemetry.md>), [tracing](<https://devfeed.tech/tags/tracing.md>)

### AI overview

OpenChoreo uses OpenTelemetry Collector pipelines to bring distributed traces and persistent Kubernetes events into its developer portal. For tracing, applications export OTLP spans; data plane Collectors attach Kubernetes and OpenChoreo metadata, apply tail sampling, then forward traces to a central Collector or supported cloud and OTLP backends. For events, a single-replica Collector watches cluster events, while a custom processor enriches them from in-memory informer caches with object labels, annotations, and owner references, avoiding API calls on the event path. Events are then exported to durable backends, so they remain available after the Kubernetes API server's roughly one-hour retention period. The portal makes traces filterable by component, environment, and time, and displays enriched events by component. OpenChoreo favors vendor-neutral components and configuration, adding custom Go code for event enrichment where existing components did not fit.

### Source excerpt

Getting code from a developer's editor into production means building it, testing it, deploying it, and then understanding how it behaves once it's running. Internal Developer Platforms (IDPs) promise a "golden path" through all of that,...

[Next page](<https://devfeed.tech/topics/tracing.md?cursor=WyIyMDI2LTEwLTI3VDExOjAwOjAwKzAwOjAwIiwgImJmZDQ3MTRlLTViMWYtNDE3ZS1hMWJkLWNiZGI2YzExM2MzYyJd>)