# telemetry

Data emitted by software systems about their behavior, including traces, metrics, and logs.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## 4 GB of Logs a Minute: A Kafka Monitoring Incident With 20 Minutes to Spare

DevFeed: [4 GB of Logs a Minute: A Kafka Monitoring Incident With 20 Minutes to Spare](<https://devfeed.tech/articles/4-gb-of-logs-a-minute-a-kafka-monitoring-incident-with-20-minutes-to-spare-146459.md>)

Original publisher: [Read original article](<https://linuxhandbook.com/blog/kafka-monitoring-incident/>)

Author: Yash Kiran Patil

Published: 2026-11-20T09:34:13Z

Content type: tutorial

Language: en

Sources: [Linux Handbook](<https://devfeed.tech/sources/linux-handbook.md>)

Topics: [Monitoring](<https://devfeed.tech/topics/monitoring.md>), [incident management](<https://devfeed.tech/topics/incident-management.md>), [infrastructure monitoring](<https://devfeed.tech/topics/infrastructure-monitoring.md>), [Logging and metrics](<https://devfeed.tech/topics/logging-and-metrics.md>), [Prometheus](<https://devfeed.tech/topics/prometheus.md>), [API Logging](<https://devfeed.tech/topics/api-logging.md>), [telemetry](<https://devfeed.tech/topics/telemetry.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [availability](<https://devfeed.tech/tags/availability.md>), [blog](<https://devfeed.tech/tags/blog.md>), [bounded](<https://devfeed.tech/tags/bounded.md>), [broker](<https://devfeed.tech/tags/broker.md>), [bug](<https://devfeed.tech/tags/bug.md>), [canary](<https://devfeed.tech/tags/canary.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [cause](<https://devfeed.tech/tags/cause.md>), [change](<https://devfeed.tech/tags/change.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [contain](<https://devfeed.tech/tags/contain.md>), [fleet](<https://devfeed.tech/tags/fleet.md>), [incident](<https://devfeed.tech/tags/incident.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [logs](<https://devfeed.tech/tags/logs.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [time](<https://devfeed.tech/tags/time.md>), [warnings](<https://devfeed.tech/tags/warnings.md>)

### AI overview

A partial JMX Exporter upgrade left Kafka brokers on versions 1.6.0 and 0.9.0; under the new scrape configuration, older agents emitted rapidly growing Jetty warnings, putting disk capacity at risk before traffic began. The account links the warnings to the mixed rollout but says their precise cause is unproven without request paths, responses, configuration, and stack traces. Operators used a supervised loop to truncate one verified active log and buy time, preserving a warning sample first; this destructive workaround does not fix exporter behavior or reclaim space from deleted-but-open files. The durable response was to standardize agent versions and restart brokers one at a time during a maintenance window, checking cluster health and exporter and Prometheus metrics after each restart. The article recommends bounded log retention, version inventory, canary checks, and measured tuning rather than assuming an endpoint or concurrency bug.

### Source excerpt

A mixed JMX Exporter fleet, rapidly growing Jetty warnings, and the difference between buying time and fixing the cause.

## Monitor Valkey clusters with OpenTelemetry metrics in Amazon ElastiCache

DevFeed: [Monitor Valkey clusters with OpenTelemetry metrics in Amazon ElastiCache](<https://devfeed.tech/articles/monitor-valkey-clusters-with-opentelemetry-metrics-in-amazon-elasticache-144311.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/database/monitor-valkey-clusters-with-opentelemetry-metrics-in-amazon-elasticache/>)

Author: Paul Ross

Published: 2026-11-19T18:39:15Z

Content type: tutorial

Language: en

Sources: [AWS Database Blog](<https://devfeed.tech/sources/aws-database-blog.md>)

Topics: [Amazon ElastiCache](<https://devfeed.tech/topics/amazon-elasticache.md>), [valkey](<https://devfeed.tech/topics/valkey.md>), [Amazon CloudWatch](<https://devfeed.tech/topics/amazon-cloudwatch.md>), [OpenTelemetry](<https://devfeed.tech/topics/opentelemetry.md>), [Grafana](<https://devfeed.tech/topics/grafana.md>), [Prometheus](<https://devfeed.tech/topics/prometheus.md>), [telemetry](<https://devfeed.tech/topics/telemetry.md>)

Tags: [alarm](<https://devfeed.tech/tags/alarm.md>), [amazon-cloudwatch](<https://devfeed.tech/tags/amazon-cloudwatch.md>), [amazon-elasticache](<https://devfeed.tech/tags/amazon-elasticache.md>), [amazon-managed-grafana](<https://devfeed.tech/tags/amazon-managed-grafana.md>), [aws-cli](<https://devfeed.tech/tags/aws-cli.md>), [aws-cloudformation](<https://devfeed.tech/tags/aws-cloudformation.md>), [insights](<https://devfeed.tech/tags/insights.md>), [intermediate-200](<https://devfeed.tech/tags/intermediate-200.md>), [latency](<https://devfeed.tech/tags/latency.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [monitor](<https://devfeed.tech/tags/monitor.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [node](<https://devfeed.tech/tags/node.md>), [opentelemetry](<https://devfeed.tech/tags/opentelemetry.md>), [query](<https://devfeed.tech/tags/query.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>), [troubleshooting](<https://devfeed.tech/tags/troubleshooting.md>), [valkey](<https://devfeed.tech/tags/valkey.md>)

### AI overview

Amazon ElastiCache for Valkey sends engine and host metrics to CloudWatch in OpenTelemetry format without an exporter or collector. The article demonstrates using ElastiCache Insights to locate a hot shard or node, then PromQL to rank nodes, inspect command latency, and alarm across all nodes as cluster membership changes. Standard monitoring publishes core metrics every 60 seconds; selected detailed metrics arrive every 15 seconds and can expose short latency spikes that a longer window smooths out. Core metrics and ElastiCache Insights are free, while selected detailed metrics, PromQL API scans, and alarms incur charges. The walkthrough requires a node-based Valkey cluster in a supported Region; its Grafana guidance covers version 13 or later, with a separate Prometheus data source approach for version 12.4.

### Source excerpt

Amazon ElastiCache for Valkey now offers OpenTelemetry metrics, detailed monitoring, ElastiCache Insights, and PromQL. This post walks through one troubleshooting path: turn on detailed monitoring, find a hot shard, query with PromQL, catch a short latency spike, and create one alarm that watches every node.

## Retain more logs for longer with Flex Logs BYOBucket

DevFeed: [Retain more logs for longer with Flex Logs BYOBucket](<https://devfeed.tech/articles/retain-more-logs-for-longer-with-flex-logs-byobucket-145232.md>)

Original publisher: [Read original article](<https://www.datadoghq.com/blog/flex-logs-bring-your-own-bucket/>)

Author: Corey Brincks; Bobby Vellanki

Published: 2026-11-19T00:00:00Z

Content type: article

Language: en

Sources: [Datadog](<https://devfeed.tech/sources/datadog-the-monitor-blog.md>)

Topics: [datadog](<https://devfeed.tech/topics/datadog.md>), [telemetry](<https://devfeed.tech/topics/telemetry.md>), [Logging and metrics](<https://devfeed.tech/topics/logging-and-metrics.md>), [audit](<https://devfeed.tech/topics/audit.md>)

Tags: [audit-logs](<https://devfeed.tech/tags/audit-logs.md>), [bucket](<https://devfeed.tech/tags/bucket.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [costs](<https://devfeed.tech/tags/costs.md>), [datadog](<https://devfeed.tech/tags/datadog.md>), [flex-logs](<https://devfeed.tech/tags/flex-logs.md>), [investigations](<https://devfeed.tech/tags/investigations.md>), [log-management](<https://devfeed.tech/tags/log-management.md>), [logs](<https://devfeed.tech/tags/logs.md>), [retention](<https://devfeed.tech/tags/retention.md>), [storage](<https://devfeed.tech/tags/storage.md>)

### AI overview

Datadog Flex Logs BYOBucket, available in Preview, lets teams store indexed logs in their own cloud bucket while keeping them searchable in Datadog. Customers pay their cloud provider for storage at their negotiated rates; Datadog charges a fixed monthly fee for compute capacity, and writes logs in compressed form. Teams control bucket retention and can keep historical logs available for investigations and audits without Datadog storage fees for that retention. Flex Logs supports Log Explorer, dashboards, log-based monitoring, and querying with Bits Chat and MCP-compatible tools; Standard Indexing is not required.

### Source excerpt

Store Flex Logs in your cloud bucket with unlimited retention to control storage costs and keep logs searchable in Datadog for investigations and audits.

## MinIO AIStor Memory Reaches GA at SC26 as Datadog Certifies AIStor for On-Premises BYOC Logs

DevFeed: [MinIO AIStor Memory Reaches GA at SC26 as Datadog Certifies AIStor for On-Premises BYOC Logs](<https://devfeed.tech/articles/minio-aistor-memory-reaches-ga-at-sc26-as-datadog-certifies-aistor-for-on-premises-byoc-logs-141144.md>)

Original publisher: [Read original article](<https://www.storagereview.com/news/minio-announces-aistor-memory-is-ga-and-datadog-byoc-lakehouse-certification>)

Author: Harold Fritts

Published: 2026-11-18T14:10:16Z

Content type: news

Language: en

Sources: [StorageReview.com](<https://devfeed.tech/sources/storagereview-com.md>)

Topics: [MinIO](<https://devfeed.tech/topics/minio.md>), [datadog](<https://devfeed.tech/topics/datadog.md>), [archive search](<https://devfeed.tech/topics/archive-search.md>), [observability pipelines](<https://devfeed.tech/topics/observability-pipelines.md>), [telemetry](<https://devfeed.tech/topics/telemetry.md>), [ai observability](<https://devfeed.tech/topics/ai-observability.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agent-context](<https://devfeed.tech/tags/agent-context.md>), [agent-memory](<https://devfeed.tech/tags/agent-memory.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [byoc](<https://devfeed.tech/tags/byoc.md>), [datadog](<https://devfeed.tech/tags/datadog.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [general-availability](<https://devfeed.tech/tags/general-availability.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [layer](<https://devfeed.tech/tags/layer.md>), [logs](<https://devfeed.tech/tags/logs.md>), [memory](<https://devfeed.tech/tags/memory.md>), [minio](<https://devfeed.tech/tags/minio.md>), [minio-aistor](<https://devfeed.tech/tags/minio-aistor.md>), [partnership](<https://devfeed.tech/tags/partnership.md>), [preview](<https://devfeed.tech/tags/preview.md>), [sc26](<https://devfeed.tech/tags/sc26.md>), [secrets](<https://devfeed.tech/tags/secrets.md>), [workspaces](<https://devfeed.tech/tags/workspaces.md>)

### AI overview

AIStor Memory is generally available, keeping agents' long-term memory, persistent workspaces, and secrets on enterprise-controlled infrastructure. MinIO says agents can reuse governed knowledge across sessions, but provides no measured quality, latency, or token-cost results. Separately, Datadog has certified AIStor as an on-premises storage foundation for BYOC logs: logs and BYOC indexing and search stay in the customer's environment, and query results reach Datadog over an outbound TLS connection. BYOC currently supports logs; trace and metric support is described as forthcoming. The text does not define what certification tested or provide sizing, ingest-rate, or query-performance data. AIStor Tables can store Iceberg data alongside logs, and MinIO says pricing is based on usable capacity without per-operation or egress charges.

### Source excerpt

MinIO announced the general availability of AIStor Memory today at SC26, moving its agent memory layer out of the technology preview we covered in July. AIStor Memory keeps long-term memory, persistent workspaces, and secrets for AI agents on infrastructure the enterprise controls. MinIO also announced a partnership with Datadog, which has certified AIStor as an The post MinIO AIStor Memory Reaches GA at SC26 as Datadog Certifies AIStor for On-Premises BYOC Logs appeared first on StorageReview.com.

## Bring Datadog BYOC to your on-premises data with MinIO AIStor

DevFeed: [Bring Datadog BYOC to your on-premises data with MinIO AIStor](<https://devfeed.tech/articles/bring-datadog-byoc-to-your-on-premises-data-with-minio-aistor-142007.md>)

Original publisher: [Read original article](<https://www.datadoghq.com/blog/datadog-byoc-minio-aistor/>)

Author: Sarjeel Yusuf; Anusha Podila

Published: 2026-11-18T00:00:00Z

Content type: article

Language: en

Sources: [Datadog](<https://devfeed.tech/sources/datadog-the-monitor-blog.md>)

Topics: [datadog](<https://devfeed.tech/topics/datadog.md>), [MinIO](<https://devfeed.tech/topics/minio.md>), [telemetry](<https://devfeed.tech/topics/telemetry.md>), [Monitoring](<https://devfeed.tech/topics/monitoring.md>), [observability pipelines](<https://devfeed.tech/topics/observability-pipelines.md>), [bits ai](<https://devfeed.tech/topics/bits-ai.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [alert](<https://devfeed.tech/tags/alert.md>), [alerting](<https://devfeed.tech/tags/alerting.md>), [api](<https://devfeed.tech/tags/api.md>), [api-key](<https://devfeed.tech/tags/api-key.md>), [applications](<https://devfeed.tech/tags/applications.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [balance](<https://devfeed.tech/tags/balance.md>), [bits](<https://devfeed.tech/tags/bits.md>), [bits-ai](<https://devfeed.tech/tags/bits-ai.md>), [bring-your-own-cloud](<https://devfeed.tech/tags/bring-your-own-cloud.md>), [byoc](<https://devfeed.tech/tags/byoc.md>), [datadog](<https://devfeed.tech/tags/datadog.md>), [log-management](<https://devfeed.tech/tags/log-management.md>), [logs](<https://devfeed.tech/tags/logs.md>), [minio](<https://devfeed.tech/tags/minio.md>), [minio-aistor](<https://devfeed.tech/tags/minio-aistor.md>), [monitor](<https://devfeed.tech/tags/monitor.md>), [on-prem](<https://devfeed.tech/tags/on-prem.md>), [on-premises](<https://devfeed.tech/tags/on-premises.md>), [search](<https://devfeed.tech/tags/search.md>), [telemetry](<https://devfeed.tech/tags/telemetry.md>)

### AI overview

Datadog BYOC for Logs pairs Datadog's in-cluster indexing and search with MinIO AIStor object storage on infrastructure the organization controls. Observability Pipelines workers collect and process logs; indexers write searchable files to an AIStor bucket, while a PostgreSQL metastore tracks their locations and background compaction merges smaller files. Storage, indexing and search capacity can scale independently. Datadog searchers initiate TLS-encrypted outbound connections, authenticated with a Datadog API key, to query the on-premises data without inbound ports or a public endpoint. Log Explorer results pass through Datadog services for display but, according to the excerpt, are not stored or cached there; the indexed dataset stays in AIStor. The excerpt also describes RBAC, consent controls for Datadog support access, and disabling log samples or excluding log fields from notifications as privacy controls. AIStor erasure coding and object tiering support retention and storage-budget tradeoffs. BYOC currently supports logs; trace and metric support is described as coming soon.

### Source excerpt

Learn how Datadog BYOC and MinIO AIStor work together to help you search, monitor, and investigate logs while keeping your telemetry on-premises.

## Protect sensitive data in your traces: new controls in LangSmith

DevFeed: [Protect sensitive data in your traces: new controls in LangSmith](<https://devfeed.tech/articles/protect-sensitive-data-in-your-traces-new-controls-in-langsmith-139469.md>)

Original publisher: [Read original article](<https://www.langchain.com/blog/protect-sensitive-data-in-your-traces-new-controls-in-langsmith>)

Author: LangChain Accounts

Published: 2026-11-17T21:31:36Z

Content type: release

Language: en

Sources: [LangChain Blog](<https://devfeed.tech/sources/langchain-blog.md>)

Topics: [sensitive data](<https://devfeed.tech/topics/sensitive-data.md>), [telemetry](<https://devfeed.tech/topics/telemetry.md>), [Data Management](<https://devfeed.tech/topics/data-management.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [about](<https://devfeed.tech/tags/about.md>), [access-control](<https://devfeed.tech/tags/access-control.md>), [access-controls](<https://devfeed.tech/tags/access-controls.md>), [added](<https://devfeed.tech/tags/added.md>), [admin](<https://devfeed.tech/tags/admin.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [agents-in-production](<https://devfeed.tech/tags/agents-in-production.md>), [all](<https://devfeed.tech/tags/all.md>), [controls](<https://devfeed.tech/tags/controls.md>), [data](<https://devfeed.tech/tags/data.md>), [help](<https://devfeed.tech/tags/help.md>), [langsmith](<https://devfeed.tech/tags/langsmith.md>), [manage](<https://devfeed.tech/tags/manage.md>), [more](<https://devfeed.tech/tags/more.md>), [sensitive-data](<https://devfeed.tech/tags/sensitive-data.md>), [teams](<https://devfeed.tech/tags/teams.md>), [trace](<https://devfeed.tech/tags/trace.md>), [traces](<https://devfeed.tech/tags/traces.md>)

### AI overview

LangSmith Enterprise adds three controls for managing access to complete production traces: admins can hide trace inputs and outputs while leaving run trees, performance statistics, and metadata visible; assign workspace-specific permissions to user groups, layered on existing RBAC and ABAC; and define trace-level ABAC policies from root run metadata. Layered together, these let teams grant, for example, support staff access to relevant customers' inputs and outputs while engineers can inspect run and performance data across traces with sensitive content hidden. Policies can be adjusted as team responsibilities change. The text says full-fidelity traces can be stored without redacted or duplicate copies; the controls determine what each user can view. These features are available on LangSmith Enterprise plans.

### Source excerpt

New controls are now in LangSmith to help teams manage access to sensitive trace data. Learn more.

## UniSage: A Unified and Diagnosis-Driven Sampling Framework for Microservice Systems

DevFeed: [UniSage: A Unified and Diagnosis-Driven Sampling Framework for Microservice Systems](<https://devfeed.tech/articles/unisage-a-unified-and-diagnosis-driven-sampling-framework-for-microservice-systems-137912.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2509.26336>)

Author: Zhouruixing Zhu, Zhihan Jiang, Tianyi Yang, Pinjia He

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [observability](<https://devfeed.tech/topics/observability.md>), [telemetry](<https://devfeed.tech/topics/telemetry.md>)

Tags: [analysis](<https://devfeed.tech/tags/analysis.md>), [budget](<https://devfeed.tech/tags/budget.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [data](<https://devfeed.tech/tags/data.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [diagnosis](<https://devfeed.tech/tags/diagnosis.md>), [diagnostic](<https://devfeed.tech/tags/diagnostic.md>), [evaluations](<https://devfeed.tech/tags/evaluations.md>), [existing](<https://devfeed.tech/tags/existing.md>), [failures](<https://devfeed.tech/tags/failures.md>), [focus](<https://devfeed.tech/tags/focus.md>), [framework](<https://devfeed.tech/tags/framework.md>), [lightweight](<https://devfeed.tech/tags/lightweight.md>), [log](<https://devfeed.tech/tags/log.md>), [logs](<https://devfeed.tech/tags/logs.md>), [measurements](<https://devfeed.tech/tags/measurements.md>), [messages](<https://devfeed.tech/tags/messages.md>), [modal](<https://devfeed.tech/tags/modal.md>), [observability](<https://devfeed.tech/tags/observability.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [systems](<https://devfeed.tech/tags/systems.md>)

### AI overview

UniSage aims to retain more useful diagnostic evidence from microservice traces and logs under storage limits. It integrates lightweight analysis into sampling decisions and jointly samples traces and logs using their cross-modal correlations, addressing the risk that sample-before-analysis discards valuable data and that independent log filtering breaks context. Across three multimodal datasets, at a 5% budget it improved average diagnostic trace recall by 6.8%-22.0% over trace-sampling baselines and had the highest diagnostic log recall on evaluated log-labeled datasets. Runtime measurements showed modest overhead across the tested workloads.

### Source excerpt

arXiv:2509.26336v3 Announce Type: replace Abstract: Traces and logs are essential observability signals for diagnosing microservice failures, but their massive volume makes full retention prohibitively expensive. Sampling is therefore widely used to reduce storage and analysis overhead. However, existing sampling frameworks suffer from two key limitations. First, they typically follow a sample-before-analysis paradigm, where data is discarded before its diagnostic value is known, risking the loss of critical evidence for downstream analysis. Second, they mainly focus on trace sampling, while logs are either compressed or filtered independently, which can break the contextual linkage between anomalous traces and their explanatory log messages. To address these limitations, we propose UniSage, a unified and diagnosis-driven sampling framework for microservice systems. UniSage is unified in two senses: it integrates lightweight analysis into the sampling pipeline to guide retention decisions, and it jointly samples traces and logs by exploiting their cross-modal correlations. Evaluations on three multi-modal microservice datasets show that UniSage consistently preserves more diagnostic evidence under tight budgets. At the 5% budget, UniSage improves average diagnostic trace recall by 6.8%-22.0% over existing trace sampling baselines and achieves the highest diagnostic log recall on the evaluated log-labeled datasets. UniSage also remains practical, with runtime measurements showing modest overhead across the evaluated workloads.

## How we scaled metrics aggregation by moving durable state into storage

DevFeed: [How we scaled metrics aggregation by moving durable state into storage](<https://devfeed.tech/articles/how-we-scaled-metrics-aggregation-by-moving-durable-state-into-storage-138310.md>)

Original publisher: [Read original article](<https://www.datadoghq.com/blog/engineering/how-we-scaled-metrics-aggregation/>)

Author: Khayyam Guliyev

Published: 2026-11-17T00:00:00Z

Content type: article

Language: en

Sources: [Datadog](<https://devfeed.tech/sources/datadog.md>), [Datadog](<https://devfeed.tech/sources/datadog-the-monitor-blog.md>)

Topics: [Logging and metrics](<https://devfeed.tech/topics/logging-and-metrics.md>), [Stateful](<https://devfeed.tech/topics/stateful.md>), [Persistence](<https://devfeed.tech/topics/persistence.md>), [sharding](<https://devfeed.tech/topics/sharding.md>), [telemetry](<https://devfeed.tech/topics/telemetry.md>), [observability](<https://devfeed.tech/topics/observability.md>)

Tags: [1-5](<https://devfeed.tech/tags/1-5.md>), [account](<https://devfeed.tech/tags/account.md>), [added](<https://devfeed.tech/tags/added.md>), [after](<https://devfeed.tech/tags/after.md>), [aggregate](<https://devfeed.tech/tags/aggregate.md>), [aggregates](<https://devfeed.tech/tags/aggregates.md>), [aggregation](<https://devfeed.tech/tags/aggregation.md>), [all](<https://devfeed.tech/tags/all.md>), [allowed](<https://devfeed.tech/tags/allowed.md>), [allows](<https://devfeed.tech/tags/allows.md>), [also](<https://devfeed.tech/tags/also.md>), [datadog](<https://devfeed.tech/tags/datadog.md>), [fixed](<https://devfeed.tech/tags/fixed.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [moving](<https://devfeed.tech/tags/moving.md>), [ownership](<https://devfeed.tech/tags/ownership.md>), [partition](<https://devfeed.tech/tags/partition.md>), [reconciliation](<https://devfeed.tech/tags/reconciliation.md>), [state](<https://devfeed.tech/tags/state.md>), [storage](<https://devfeed.tech/tags/storage.md>), [tags](<https://devfeed.tech/tags/tags.md>)

### AI overview

Datadog's metrics aggregation scaled through three designs as durable state became a bottleneck. The first used fixed Kafka partition ownership, with leader and follower instances holding hours of RocksDB state; it provided failover but duplicated compute and required manual, risky repartitioning. The second dynamically reassigned partitions and uploaded incremental state to blob storage, restoring recent windows first and older state when late data required it. This enabled autoscaling and about 10x more volume, but recovery and state movement eventually constrained scale. The third makes streaming aggregation and deduplication best-effort: workers prioritize on-time traffic and can reset to the live edge, while storage reconciles partial aggregates, duplicates, late data, and backfills across its recent and long-term tiers. This removes hours of durable local state and snapshot recovery from the streaming path, while placing durable correctness in storage.

### Source excerpt

Learn how Datadog evolved metrics aggregation from fixed partition ownership to transferable state and, ultimately, storage-side reconciliation.

## Get useful logs from any source with Datadog Auto-Processing

DevFeed: [Get useful logs from any source with Datadog Auto-Processing](<https://devfeed.tech/articles/get-useful-logs-from-any-source-with-datadog-auto-processing-136095.md>)

Original publisher: [Read original article](<https://www.datadoghq.com/blog/auto-processing-log-pipelines/>)

Author: Nicolas Jung; Corey Brincks

Published: 2026-11-16T00:00:00Z

Content type: article

Language: en

Sources: [Datadog](<https://devfeed.tech/sources/datadog-the-monitor-blog.md>)

Topics: [observability pipelines](<https://devfeed.tech/topics/observability-pipelines.md>), [telemetry](<https://devfeed.tech/topics/telemetry.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [automate](<https://devfeed.tech/tags/automate.md>), [automatic](<https://devfeed.tech/tags/automatic.md>), [automatically](<https://devfeed.tech/tags/automatically.md>), [core](<https://devfeed.tech/tags/core.md>), [custom](<https://devfeed.tech/tags/custom.md>), [dashboard](<https://devfeed.tech/tags/dashboard.md>), [dashboards](<https://devfeed.tech/tags/dashboards.md>), [data](<https://devfeed.tech/tags/data.md>), [datadog](<https://devfeed.tech/tags/datadog.md>), [formats](<https://devfeed.tech/tags/formats.md>), [keep](<https://devfeed.tech/tags/keep.md>), [log-management](<https://devfeed.tech/tags/log-management.md>), [logs](<https://devfeed.tech/tags/logs.md>), [parsing](<https://devfeed.tech/tags/parsing.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [processing](<https://devfeed.tech/tags/processing.md>), [rules](<https://devfeed.tech/tags/rules.md>), [source](<https://devfeed.tech/tags/source.md>), [structure](<https://devfeed.tech/tags/structure.md>), [structured](<https://devfeed.tech/tags/structured.md>), [updating](<https://devfeed.tech/tags/updating.md>)

### AI overview

Datadog Auto-Processing automatically detects custom logs without an applicable pipeline, generates parsing rules, and applies them when confidence and risk scores meet a required threshold; the process takes about an hour. It extracts queryable values and maps core fields such as timestamp, status, service, trace ID, and span ID, supporting consistent searches, dashboards, metrics, and monitors across different log formats. It also adapts pipelines it manages when formats drift, helping preserve attributes used by downstream workflows. Pipeline creation requires no manual initiation, rule writing, or approval. The excerpt does not specify how users can review or control generated rules.

### Source excerpt

Use Auto-Processing to structure logs with custom formats and keep them structured without manually building pipelines or updating parsing rules.

## Your Self-Hosted Apps May Be Sending Data You Didn't Know About

DevFeed: [Your Self-Hosted Apps May Be Sending Data You Didn't Know About](<https://devfeed.tech/articles/your-self-hosted-apps-may-be-sending-data-you-didn-t-know-about-134311.md>)

Original publisher: [Read original article](<https://www.virtualizationhowto.com/2026/11/your-self-hosted-apps-may-be-sending-data-you-didnt-know-about/>)

Author: Brandon Lee

Published: 2026-11-14T13:56:33Z

Content type: article

Language: en

Sources: [Virtualization Howto](<https://devfeed.tech/sources/virtualization-howto.md>)

Topics: [Self-hosted](<https://devfeed.tech/topics/self-hosted.md>), [Homelab](<https://devfeed.tech/topics/homelab.md>), [telemetry](<https://devfeed.tech/topics/telemetry.md>)

Tags: [compose](<https://devfeed.tech/tags/compose.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [containers](<https://devfeed.tech/tags/containers.md>), [controls](<https://devfeed.tech/tags/controls.md>), [data-leak](<https://devfeed.tech/tags/data-leak.md>), [data-privacy](<https://devfeed.tech/tags/data-privacy.md>), [database](<https://devfeed.tech/tags/database.md>), [home-lab](<https://devfeed.tech/tags/home-lab.md>), [home-server](<https://devfeed.tech/tags/home-server.md>), [self-hosted](<https://devfeed.tech/tags/self-hosted.md>)

### AI overview

Immich's machine learning container made repeated requests to a Microsoft telemetry endpoint because ONNX Runtime 1.29 enabled telemetry by default on certain Linux builds. The article says the data did not include photos or videos; reported telemetry could include persistent device identifiers, OS details, CPU, RAM, and the machine learning execution provider. Immich 3.3.6 disables this telemetry; older deployments can set ORT_DISABLE_TELEMETRY="1" for the machine learning service and recreate its container. The article stresses that this setting only applies to ONNX Runtime. To investigate other unexpected egress, it suggests reviewing DNS query logs, inspecting Docker networks, and checking a container's connections with ss in its network namespace via nsenter; connection checks show only a momentary snapshot. Docker Compose networks can be marked internal: true to restrict internet access, though that can break apps needing external services or model downloads. The author also advocates blocking outbound traffic by default and selectively allowing required connections.

### Source excerpt

I would say that one of the biggest reasons that many self-host instead of put things in the cloud is data privacy and having control over their data. But a... The post Your Self-Hosted Apps May Be Sending Data You Didn't Know About appeared first on Virtualization Howto.

## How Red Hat OpenShift 5 simplifies incident diagnosis with supervised AI

DevFeed: [How Red Hat OpenShift 5 simplifies incident diagnosis with supervised AI](<https://devfeed.tech/articles/how-red-hat-openshift-5-simplifies-incident-diagnosis-with-supervised-ai-129635.md>)

Original publisher: [Read original article](<https://www.redhat.com/en/blog/how-red-hat-openshift-5-simplifies-incident-diagnosis-supervised-ai>)

Author: Vanessa Martini

Published: 2026-11-11T00:00:00Z

Content type: article

Language: en

Sources: [Red Hat Blog](<https://devfeed.tech/sources/red-hat-blog.md>)

Topics: [AIOps for Kubernetes](<https://devfeed.tech/topics/aiops-for-kubernetes.md>), [observability](<https://devfeed.tech/topics/observability.md>), [telemetry](<https://devfeed.tech/topics/telemetry.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [human-in-the-loop UX](<https://devfeed.tech/topics/human-in-the-loop-ux.md>), [Prometheus](<https://devfeed.tech/topics/prometheus.md>), [OpenTelemetry](<https://devfeed.tech/topics/opentelemetry.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [alert](<https://devfeed.tech/tags/alert.md>), [alert-fatigue](<https://devfeed.tech/tags/alert-fatigue.md>), [cluster](<https://devfeed.tech/tags/cluster.md>), [data](<https://devfeed.tech/tags/data.md>), [diagnosis](<https://devfeed.tech/tags/diagnosis.md>), [engineer](<https://devfeed.tech/tags/engineer.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [incident](<https://devfeed.tech/tags/incident.md>), [language](<https://devfeed.tech/tags/language.md>), [logs](<https://devfeed.tech/tags/logs.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [openshift](<https://devfeed.tech/tags/openshift.md>), [openshift-5](<https://devfeed.tech/tags/openshift-5.md>), [platform](<https://devfeed.tech/tags/platform.md>), [prometheus](<https://devfeed.tech/tags/prometheus.md>), [red-hat-openshift](<https://devfeed.tech/tags/red-hat-openshift.md>), [security-governance](<https://devfeed.tech/tags/security-governance.md>), [telemetry](<https://devfeed.tech/tags/telemetry.md>), [traces](<https://devfeed.tech/tags/traces.md>)

### AI overview

OpenShift Lightspeed turns alert response into structured, human-reviewed investigation: it can correlate Prometheus metrics, pod logs, and installed OpenTelemetry traces into evidence, signals, and draft findings. On OpenShift 4.22+ with Lightspeed operator 1.1.5+, teams can also request Perses metric panels in natural language; with cluster observability operator 1.6, panels are marked AI-assisted and expose PromQL, time range, and source metrics for review. OpenShift 5 adds an opt-in technology preview that watches existing Alertmanager receivers, runs standard oc/kubectl diagnostics, and stages remediation for approval. It requires Lightspeed operator 2 and a configured tool-calling LLM; no default model ships. Changes require administrator approval, run in ephemeral RBAC-sandboxed environments, and are audited, with failed remediation halted and escalated. Model endpoint choice determines where data is processed; locally hosted models can support air-gapped use.

### Source excerpt

Prometheus metrics, traces, and logs are rarely missing in modern enterprise environments. Most platform teams already have plenty of telemetry at their disposal. The real bottleneck is the sheer volume of data a human must parse before taking meaningful action during an incident.On a busy cluster, this creates a familiar and exhausting pattern: an alert fires, an engineer opens 3 different consoles, brings up a Prometheus Query Language (PromQL) scratchpad, executes a log query, and spends 20 minutes debating whether the signal represents a genuine emergency or background noise. Alert fatigue

## Make Kubernetes logs easier to correlate with traces using the log normalizer operator

DevFeed: [Make Kubernetes logs easier to correlate with traces using the log normalizer operator](<https://devfeed.tech/articles/make-kubernetes-logs-easier-to-correlate-with-traces-using-the-log-normalizer-operator-121178.md>)

Original publisher: [Read original article](<https://developers.redhat.com/articles/2026/11/06/make-kubernetes-logs-easier-to-correlate-with-traces-using-the-log-normalizer-operator>)

Author: Denis Moskalenko

Published: 2026-11-06T13:01:47Z

Content type: article

Language: en

Sources: [Red Hat](<https://devfeed.tech/sources/red-hat.md>), [Red Hat Developer](<https://devfeed.tech/sources/red-hat-developer.md>)

Topics: [log management](<https://devfeed.tech/topics/log-management.md>), [telemetry](<https://devfeed.tech/topics/telemetry.md>), [observability pipelines](<https://devfeed.tech/topics/observability-pipelines.md>), [OpenTelemetry](<https://devfeed.tech/topics/opentelemetry.md>), [kubectl](<https://devfeed.tech/topics/kubectl.md>)

Tags: [collector](<https://devfeed.tech/tags/collector.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [logs](<https://devfeed.tech/tags/logs.md>), [observability](<https://devfeed.tech/tags/observability.md>), [openshift](<https://devfeed.tech/tags/openshift.md>), [opentelemetry](<https://devfeed.tech/tags/opentelemetry.md>), [opentelemetry-collector](<https://devfeed.tech/tags/opentelemetry-collector.md>), [operator](<https://devfeed.tech/tags/operator.md>), [red-hat-openshift](<https://devfeed.tech/tags/red-hat-openshift.md>), [sidecar](<https://devfeed.tech/tags/sidecar.md>), [traces](<https://devfeed.tech/tags/traces.md>)

### AI overview

The Log Normalizer Operator standardizes heterogeneous Kubernetes logs at pod admission, enabling correlation with traces without changing application code. A namespace selector triggers a webhook to inject a sidecar and wrapper; the wrapper tees stdout and stderr through a named pipe, and the sidecar detects common formats, reassembles multiline entries, and emits structured JSON with fields such as timestamp, level, message, trace and span IDs, and Kubernetes identity. Records remain available through Kubernetes logs and can optionally go to an OpenTelemetry collector over OTLP; Korrel8r can use shared context, but its configuration and backends remain deployment-specific. The deployment model requires native sidecar support (Kubernetes 1.33+ or OpenShift 4.18+); existing pods need recreation, and failed image entrypoint resolution can leave a container unwrapped. The webhook's Ignore failure policy lets pods start without injection during outages. The sidecar complements rather than replaces the collector. Reported resource and latency measurements are workload-specific, and the operator does not define production redaction or sampling policies.

### Source excerpt

The log normalizer operator for Red Hat OpenShift injects a sidecar into selected pods, captures application output without requiring application code changes, detects the input format, reassembles multiline entries, and emits a consistent structured record. The normalized record remains available through kubectl logs and can also be exported to an OpenTelemetry collector over OTLP. The post Make Kubernetes logs easier to correlate with traces using the log normalizer operator appeared first on Red Hat Developer.

## Fast Unsupervised Anomaly Detection Methods on Software Logs - An Empirical Study

DevFeed: [Fast Unsupervised Anomaly Detection Methods on Software Logs - An Empirical Study](<https://devfeed.tech/articles/fast-unsupervised-anomaly-detection-methods-on-software-logs-an-empirical-study-120824.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.07760>)

Author: Jesse Nyyss\\"ol\\"a, Mika M\\"antyl\\"a, Yuqing Wang

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [threat detection](<https://devfeed.tech/topics/threat-detection.md>), [Data analysis](<https://devfeed.tech/topics/data-analysis.md>), [reliability-engineering](<https://devfeed.tech/topics/reliability-engineering.md>), [clustering](<https://devfeed.tech/topics/clustering.md>), [telemetry](<https://devfeed.tech/topics/telemetry.md>), [Hadoop](<https://devfeed.tech/topics/hadoop.md>), [hdfs](<https://devfeed.tech/topics/hdfs.md>)

Tags: [anomaly-detection](<https://devfeed.tech/tags/anomaly-detection.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [data-preprocessing](<https://devfeed.tech/tags/data-preprocessing.md>), [fast](<https://devfeed.tech/tags/fast.md>), [logs](<https://devfeed.tech/tags/logs.md>), [software](<https://devfeed.tech/tags/software.md>)

### AI overview

The study finds that unsupervised log anomaly detection results vary substantially across datasets, so no method or preprocessing choice works best universally. Across seven datasets, four models and three representations, it measures AUCROC, F1 and execution time. Log parsing helps on some datasets and hurts on others; anomaly-free training data may improve performance but takes extra effort to identify. In the tested unsupervised setup, randomized training data did not produce the data-leakage effect reported for supervised models. For faster execution, the abstract recommends word representations and avoiding IsolationForest.

### Source excerpt

arXiv:2611.07760v1 Announce Type: new Abstract: Anomaly detection on software logs ensures system reliability through failure identification, root cause analysis and fault localization. While supervised models show excellent performance, in a practical setting there rarely are labeled training data available. This calls for the use of unsupervised models. We study effects of data preprocessing, anomaly detection method, and the dataset. For preprocessing, we examine three options: The use of anomaly-free training data, using randomized as opposed to chronologically selected training data, and the use of a custom loader to extract log message from the log line. Regarding anomaly detection we use four models (KMeans, IsolationForest, RarityModel, Out-Of-Vocabulary Detector) and three representations (Words, trigrams, and parsed events). We evaluate these approaches on seven datasets: BGL, HDFS, Hadoop, Spirit, Thunderbird, Liberty, and LO2Token. We run each combination of the selected factors and measure performance with AUCROC and F1-score metrics. Then we utilize a generalized linear model to establish relationships between the factors. Furthermore, we address execution times. Our evidence reveals large variability across all datasets, preventing us from making recommendations that apply universally. For instance, log parsing enhances performance on some datasets but reduces it on others. The evidence suggests that using anomaly-free data for training unsupervised models can improve anomaly detection performance, though this requires extra effort to identify such data.Previous work has argued that shuffling data leads to unrealistic model performance (data leakage). Yet, our results show this does not hold for the unsupervised models we tested, while it remains true for supervised models in the same setup. For fast execution of the pipeline, use word representation and avoid IsolationForest model.

## Usage metrics API flags telemetry and version gaps

DevFeed: [Usage metrics API flags telemetry and version gaps](<https://devfeed.tech/articles/usage-metrics-api-flags-telemetry-and-version-gaps-120337.md>)

Original publisher: [Read original article](<https://github.blog/changelog/2026-11-05-usage-metrics-api-flags-telemetry-and-version-gaps>)

Author: Allison

Published: 2026-11-06T00:08:31Z

Content type: article

Language: en

Sources: [GitHub Changelog](<https://devfeed.tech/sources/github-changelog.md>)

Topics: [telemetry](<https://devfeed.tech/topics/telemetry.md>), [API](<https://devfeed.tech/topics/api.md>), [Monitoring](<https://devfeed.tech/topics/monitoring.md>), [version](<https://devfeed.tech/topics/version.md>)

Tags: [account-management](<https://devfeed.tech/tags/account-management.md>), [api](<https://devfeed.tech/tags/api.md>), [billing](<https://devfeed.tech/tags/billing.md>), [client](<https://devfeed.tech/tags/client.md>), [copilot](<https://devfeed.tech/tags/copilot.md>), [documentation](<https://devfeed.tech/tags/documentation.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [enterprise-management-tools](<https://devfeed.tech/tags/enterprise-management-tools.md>), [events](<https://devfeed.tech/tags/events.md>), [extension](<https://devfeed.tech/tags/extension.md>), [feature](<https://devfeed.tech/tags/feature.md>), [firewalls](<https://devfeed.tech/tags/firewalls.md>), [ide](<https://devfeed.tech/tags/ide.md>), [improvement](<https://devfeed.tech/tags/improvement.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [model](<https://devfeed.tech/tags/model.md>), [organization](<https://devfeed.tech/tags/organization.md>), [permission](<https://devfeed.tech/tags/permission.md>), [reports](<https://devfeed.tech/tags/reports.md>), [telemetry](<https://devfeed.tech/tags/telemetry.md>), [user](<https://devfeed.tech/tags/user.md>), [users](<https://devfeed.tech/tags/users.md>), [version](<https://devfeed.tech/tags/version.md>)

### AI overview

The Copilot usage metrics API adds signals to identify users whose activity lacks editor telemetry or whose IDEs or Copilot extensions fall below minimum supported versions. Enterprise and organization user reports include per-user `client_telemetry_present` and `usage_metrics_requirements` fields; reports also provide daily counts of affected users. Missing telemetry users still count as active, but their reports omit details such as lines of code and feature or model breakdowns. Telemetry and version compliance are independent signals. The daily counts can overlap and should not be added together or across days to estimate unique users. Access requires the Copilot usage metrics policy to be enabled and an eligible owner, billing manager, or role with `View Copilot Metrics` permission. Null or omitted values indicate unavailable data, including for dates before the release.

### Source excerpt

The Copilot usage metrics API now shows which users are missing client telemetry or are running IDE or extension versions below the usage metrics minimums. New per-user fields appear in... The post Usage metrics API flags telemetry and version gaps appeared first on The GitHub Blog.

## Stream your customers' audit logs into their own Snowflake warehouse

DevFeed: [Stream your customers' audit logs into their own Snowflake warehouse](<https://devfeed.tech/articles/stream-your-customers-audit-logs-into-their-own-snowflake-warehouse-121079.md>)

Original publisher: [Read original article](<https://workos.com/blog/audit-log-streaming-snowflake>)

Author: WorkOS

Published: 2026-11-05T00:00:00Z

Content type: tutorial

Language: en

Sources: [WorkOS](<https://devfeed.tech/sources/workos-blog.md>)

Topics: [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>), [audit](<https://devfeed.tech/topics/audit.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [telemetry](<https://devfeed.tech/topics/telemetry.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [2026](<https://devfeed.tech/tags/2026.md>), [3](<https://devfeed.tech/tags/3.md>), [audit](<https://devfeed.tech/tags/audit.md>), [audit-logs](<https://devfeed.tech/tags/audit-logs.md>), [auth](<https://devfeed.tech/tags/auth.md>), [customer](<https://devfeed.tech/tags/customer.md>), [customers](<https://devfeed.tech/tags/customers.md>), [event](<https://devfeed.tech/tags/event.md>), [fixed](<https://devfeed.tech/tags/fixed.md>), [key](<https://devfeed.tech/tags/key.md>), [logs](<https://devfeed.tech/tags/logs.md>), [pair](<https://devfeed.tech/tags/pair.md>), [schema](<https://devfeed.tech/tags/schema.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>), [stream](<https://devfeed.tech/tags/stream.md>), [table](<https://devfeed.tech/tags/table.md>), [up](<https://devfeed.tech/tags/up.md>), [warehouse](<https://devfeed.tech/tags/warehouse.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

WorkOS Log Streams can write each organization's audit events as rows in the customer's own Snowflake account, where the customer controls storage and retention. Setup requires a dedicated user authenticated by an RSA key pair, a table with the documented schema, and a default role granted access to the warehouse, database, schema, and table; accounts with network policies must allow WorkOS's published IP addresses. The dashboard and Admin Portal configure the same stream, but the portal lets the customer provide credentials directly so the private key need not pass through support. Monitor actual delivery with a customer-specific freshness query in Snowflake: an Active state alone does not establish that rows are arriving, and the article says not to assume stop-delivery notifications. Error or Invalid streams require fixing the cause and revalidating. Delivery resumes from the last successful event, subject to the organization's WorkOS retention window; previously delivered rows remain subject to the customer's own warehouse policy.

### Source excerpt

Each audit event lands as a row in your customer's own Snowflake table, with key-pair auth, a fixed schema, grants up front, and a freshness check.

## Trace linkage: connect the dots between your app traces and database queries

DevFeed: [Trace linkage: connect the dots between your app traces and database queries](<https://devfeed.tech/articles/trace-linkage-connect-the-dots-between-your-app-traces-and-database-queries-116281.md>)

Original publisher: [Read original article](<https://grafana.com/blog/trace-linkage-connect-the-dots-between-your-app-traces-and-database-queries/>)

Author: Shirley

Published: 2026-11-03T15:05:03Z

Content type: release

Language: en

Sources: [Grafana Labs](<https://devfeed.tech/sources/grafana-labs-blog-on-grafana-labs.md>)

Topics: [observability](<https://devfeed.tech/topics/observability.md>), [OpenTelemetry](<https://devfeed.tech/topics/opentelemetry.md>), [telemetry](<https://devfeed.tech/topics/telemetry.md>), [Grafana Alloy](<https://devfeed.tech/topics/grafana-alloy.md>), [Grafana Cloud](<https://devfeed.tech/topics/grafana-cloud.md>)

Tags: [alloy](<https://devfeed.tech/tags/alloy.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [application](<https://devfeed.tech/tags/application.md>), [database](<https://devfeed.tech/tags/database.md>), [database-observability](<https://devfeed.tech/tags/database-observability.md>), [execution](<https://devfeed.tech/tags/execution.md>), [explain](<https://devfeed.tech/tags/explain.md>), [grafana](<https://devfeed.tech/tags/grafana.md>), [grafana-cloud](<https://devfeed.tech/tags/grafana-cloud.md>), [history](<https://devfeed.tech/tags/history.md>), [logs](<https://devfeed.tech/tags/logs.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [observability](<https://devfeed.tech/tags/observability.md>), [opentelemetry](<https://devfeed.tech/tags/opentelemetry.md>), [performance](<https://devfeed.tech/tags/performance.md>), [plan](<https://devfeed.tech/tags/plan.md>), [queries](<https://devfeed.tech/tags/queries.md>), [telemetry](<https://devfeed.tech/tags/telemetry.md>), [tempo](<https://devfeed.tech/tags/tempo.md>), [text](<https://devfeed.tech/tags/text.md>), [trace](<https://devfeed.tech/tags/trace.md>), [traces](<https://devfeed.tech/tags/traces.md>), [tracing](<https://devfeed.tech/tags/tracing.md>)

### AI overview

Grafana Cloud Database Observability's trace linkage connects OpenTelemetry database spans to query context collected by Alloy. It uses query text, database system, and database name; spans with `db.query.text` or legacy `db.statement` may match without further application instrumentation. For more precise execution-level matches, optional sqlcommenter propagation places trace and span IDs in SQL comments, which Alloy extracts from query sample logs. Exact matches are best-effort: Alloy may miss short-lived executions or fail to poll them, sqlcommenter does not support every database engine and may affect database caches. When exact matching is unavailable, trace linkage accounts for differences in query formatting, comments, literals, and parameter placeholders to find matching query text, allowing users to inspect samples, historical performance, schema, and explain plans. Setup requires an OpenTelemetry-instrumented application sending traces to Grafana Cloud and Database Observability configured.

### Source excerpt

When an application request slows down, a trace can show a database call. Finding the database query behind it can be tedious, requiring you to switch between traces and database telemetry, as well as match timestamps or query text by hand. In Grafana Cloud, the Database Observability app already provides a window into all of your database telemetry, including query samples, performance history, and explain plan. But what if you could jump from that database call in the trace straight to its query details? OpenTelemetry database semantic conventions stabilized last year, giving us all a dependable shared language for describing database operations. That foundation has opened the door for what we call trace linkage, now Generally Available in Database Observability! What is trace linkage? With trace linkage, we try to provide a connection between OpenTelemetry-instrumented applications and database telemetry collected with Alloy. In other words, when you give us these previously-disjoint puzzle pieces, we'll do our best to provide you as much supporting information between these two elements as we can. And we'll do it without making you tediously piece together timestamps and logs. From a database span in a trace, you'll see a link on the query text. Clicking from there, the app searches for a connection between your database query and our opinionated view of your most relevant data in the Database Observability app. You can explore sampled query execution data or check out the query's historical performance, alongside that query's related schema and explain plan. This connection builds on the database attributes your OpenTelemetry instrumentation already emits. Grafana Cloud Traces, powered by Tempo, retain those attributes, and trace linkage uses the query text, database system, and database name to connect a database span to its supporting query context in Database Observability. We support the stable conventions like db.query.text and its legacy equivalent db.sta

## Finding performance wins in our own traces with a coding agent

DevFeed: [Finding performance wins in our own traces with a coding agent](<https://devfeed.tech/articles/finding-performance-wins-in-our-own-traces-with-a-coding-agent-115871.md>)

Original publisher: [Read original article](<https://blog.sentry.io/performance-wins-traces-coding-agent/>)

Author: Scott Cooper

Published: 2026-11-03T09:00:00Z

Content type: news

Language: en

Sources: [Sentry](<https://devfeed.tech/sources/sentry-blog.md>)

Topics: [database-optimization](<https://devfeed.tech/topics/database-optimization.md>), [telemetry](<https://devfeed.tech/topics/telemetry.md>), [dogfooding](<https://devfeed.tech/topics/dogfooding.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [backend](<https://devfeed.tech/tags/backend.md>), [coding](<https://devfeed.tech/tags/coding.md>), [coding-agent](<https://devfeed.tech/tags/coding-agent.md>), [mcp](<https://devfeed.tech/tags/mcp.md>), [performance](<https://devfeed.tech/tags/performance.md>), [queries](<https://devfeed.tech/tags/queries.md>), [sentry](<https://devfeed.tech/tags/sentry.md>), [traces](<https://devfeed.tech/tags/traces.md>)

### AI overview

Using Claude with Opus 5.5 and the Sentry MCP, the author ranked frequent, slow spans in Sentry's own traces, then investigated selected issues in code and git history before proposing small fixes as draft PRs. Django query spans, query-source attributes, and example trace links helped locate and review work. For SQL changes, the agent compared EXPLAIN ANALYZE results on a read-only database mirror. Reported wins included about 143 million fewer SELECTs per week from reusing loaded project data, a GroupHistory lookup reduced from 935 ms to 0.4 ms on a warm cache, and batched cache writes reduced from about 51 ms to 1.2 ms in a measured example. The author also checked request latency and database load, and says query-volume and daily-savings figures are estimates based on sampled spans and observed traffic.

### Source excerpt

How we used a coding agent and the Sentry MCP to find backend performance wins in Sentry's own traces and remove hundreds of millions of queries a week.

## OpenTelemetry eBPF Instrumentation reaches v1.0.0

DevFeed: [OpenTelemetry eBPF Instrumentation reaches v1.0.0](<https://devfeed.tech/articles/opentelemetry-ebpf-instrumentation-reaches-v1-0-0-109296.md>)

Original publisher: [Read original article](<https://opentelemetry.io/blog/2026/obi-v1-release/>)

Author: OpenTelemetry Authors; Docs CC BY

Published: 2026-10-30T15:42:24Z

Content type: release

Language: en

Sources: [OpenTelemetry](<https://devfeed.tech/sources/blog-on-opentelemetry.md>)

Topics: [OpenTelemetry](<https://devfeed.tech/topics/opentelemetry.md>), [eBPF](<https://devfeed.tech/topics/ebpf.md>), [telemetry](<https://devfeed.tech/topics/telemetry.md>), [tracing](<https://devfeed.tech/topics/tracing.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-applications](<https://devfeed.tech/tags/ai-applications.md>), [alerts](<https://devfeed.tech/tags/alerts.md>), [announce](<https://devfeed.tech/tags/announce.md>), [applications](<https://devfeed.tech/tags/applications.md>), [compatibility](<https://devfeed.tech/tags/compatibility.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [dependencies](<https://devfeed.tech/tags/dependencies.md>), [ebpf](<https://devfeed.tech/tags/ebpf.md>), [instrumentation](<https://devfeed.tech/tags/instrumentation.md>), [linux](<https://devfeed.tech/tags/linux.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [migration](<https://devfeed.tech/tags/migration.md>), [observability](<https://devfeed.tech/tags/observability.md>), [opentelemetry](<https://devfeed.tech/tags/opentelemetry.md>), [release](<https://devfeed.tech/tags/release.md>), [telemetry](<https://devfeed.tech/tags/telemetry.md>), [traces](<https://devfeed.tech/tags/traces.md>), [v1](<https://devfeed.tech/tags/v1.md>)

### AI overview

OBI 1.0 provides a stable foundation for collecting OpenTelemetry metrics and traces from running Linux applications without source changes, using eBPF and runtime-specific instrumentation. It documents compatibility commitments for Config v2, deployment interfaces, and supported environments, plus a telemetry migration path using Weaver-compatible schemas, release notes, and guidance. Config v1 remains supported for backward compatibility but is frozen; existing users are advised to migrate, review environment-based configuration, and compare telemetry in a canary before broad rollout. The release expands protocol, framework, runtime, and AI instrumentation, though coverage and capabilities depend on documented support and configuration. Runtime metrics remain experimental, and detailed AI payload capture requires explicit configuration. OBI can complement language agents and manual instrumentation when application-specific context is needed.

### Source excerpt

We're excited to announce v1.0.0 of OpenTelemetry eBPF Instrumentation--OBI! OBI brings OpenTelemetry observability to running Linux applications without requiring changes to their source code. It uses eBPF, together with runtime-specific instrumentation, to capture application metrics and traces across services, languages, and frameworks. You can start with request rates, errors, latency, and service dependencies, then build deeper visibility as your needs grow. This release fulfills the flagship goal we set for 2026: establishing a stable foundation for OBI. It represents a year of expanding instrumentation coverage, improving correctness, strengthening release tooling, and building the compatibility and telemetry migration practices that users need to operate OBI over time.

## Connecting Cloudflare and Honeybadger with OpenTelemetry

DevFeed: [Connecting Cloudflare and Honeybadger with OpenTelemetry](<https://devfeed.tech/articles/connecting-cloudflare-and-honeybadger-with-opentelemetry-109204.md>)

Original publisher: [Read original article](<https://www.honeybadger.io/blog/cloudflare-honeybadger-otel/>)

Author: Addison Curtis

Published: 2026-10-29T07:00:00Z

Content type: tutorial

Language: en

Sources: [Honeybadger](<https://devfeed.tech/sources/honeybadger.md>)

Topics: [OpenTelemetry](<https://devfeed.tech/topics/opentelemetry.md>), [Cloudflare Workers](<https://devfeed.tech/topics/cloudflare-workers.md>), [observability pipelines](<https://devfeed.tech/topics/observability-pipelines.md>), [telemetry](<https://devfeed.tech/topics/telemetry.md>), [observability](<https://devfeed.tech/topics/observability.md>)

Tags: [cloudflare](<https://devfeed.tech/tags/cloudflare.md>), [cloudflare-workers](<https://devfeed.tech/tags/cloudflare-workers.md>), [devops-articles](<https://devfeed.tech/tags/devops-articles.md>), [honeybadger](<https://devfeed.tech/tags/honeybadger.md>), [insights](<https://devfeed.tech/tags/insights.md>), [logs](<https://devfeed.tech/tags/logs.md>), [observability](<https://devfeed.tech/tags/observability.md>), [opentelemetry](<https://devfeed.tech/tags/opentelemetry.md>)

### AI overview

The article explains how to send Cloudflare Workers logs to Honeybadger Insights over OpenTelemetry Protocol (OTLP). In Cloudflare, create a Logs destination with endpoint `https://api.honeybadger.io/v1/logs` and add the project API key as a custom `X-API-Key` header; the header name must match exactly. The author tests the pipeline by logging from a Worker route, checking Cloudflare for the event, then allowing a few minutes for it to reach Insights. The article says Cloudflare's built-in observability may suffice for small apps, while its split between logs and traces and fragmented query builder can make analysis harder for larger applications. Insights instead presents wide events and offers querying, visualizations, natural-language search, and alarms. The example uses RedwoodSDK and requires a `workers.dev` subdomain; it focuses on logs, although the article notes that Cloudflare supports OTel traces and logs and Insights supports OTLP traces, metrics, and logs.

### Source excerpt

Cloudflare's built-in observability works fine for small apps, but the search is clunky and your data gets split across events and traces. Here's how to pipe Cloudflare Workers logs into Honeybadger Insights with OpenTelemetry instead.

## Elastic Cloud Serverless audit logs: Ship the trail to Observability or Security

DevFeed: [Elastic Cloud Serverless audit logs: Ship the trail to Observability or Security](<https://devfeed.tech/articles/elastic-cloud-serverless-audit-logs-ship-the-trail-to-observability-or-security-106464.md>)

Original publisher: [Read original article](<https://www.elastic.co/search-labs/blog/elastic-cloud-serverless-audit-logs>)

Author: Valentin Crettaz

Published: 2026-10-29T00:00:00Z

Content type: article

Language: en

Sources: [Elasticsearch Labs](<https://devfeed.tech/sources/elasticsearch-labs.md>)

Topics: [Elastic Cloud](<https://devfeed.tech/topics/elastic-cloud.md>), [Logging](<https://devfeed.tech/topics/logging.md>), [Threat monitoring](<https://devfeed.tech/topics/threat-monitoring.md>), [observability pipelines](<https://devfeed.tech/topics/observability-pipelines.md>), [Security](<https://devfeed.tech/topics/security.md>), [observability](<https://devfeed.tech/topics/observability.md>), [telemetry](<https://devfeed.tech/topics/telemetry.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [alerting](<https://devfeed.tech/tags/alerting.md>), [attacks](<https://devfeed.tech/tags/attacks.md>), [audit](<https://devfeed.tech/tags/audit.md>), [audit-logs](<https://devfeed.tech/tags/audit-logs.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [dashboards](<https://devfeed.tech/tags/dashboards.md>), [detection](<https://devfeed.tech/tags/detection.md>), [elastic](<https://devfeed.tech/tags/elastic.md>), [elastic-cloud](<https://devfeed.tech/tags/elastic-cloud.md>), [elastic-cloud-serverless](<https://devfeed.tech/tags/elastic-cloud-serverless.md>), [forensics](<https://devfeed.tech/tags/forensics.md>), [kibana](<https://devfeed.tech/tags/kibana.md>), [logs](<https://devfeed.tech/tags/logs.md>), [object](<https://devfeed.tech/tags/object.md>), [observability](<https://devfeed.tech/tags/observability.md>), [operations](<https://devfeed.tech/tags/operations.md>), [rules](<https://devfeed.tech/tags/rules.md>), [security](<https://devfeed.tech/tags/security.md>), [serverless](<https://devfeed.tech/tags/serverless.md>), [space](<https://devfeed.tech/tags/space.md>)

### AI overview

Elastic Cloud Serverless log delivery is generally available for shipping a source project's Elasticsearch and Kibana activity to an Observability or Security project in the same organization. Multiple sources can share a destination; ingest and retention are billed there, and retention is configurable. Five Ignore filters can drop matching events before delivery to reduce volume and cost. The stream covers saved-object changes, index access, sign-ins and access failures, with project and producer fields for attribution. Eight included alert rules watch for credential attack patterns; the first-seen identity rule needs at least a day of history. Seven dashboards include Kibana Object Forensics, Elasticsearch Data Access and Identity & Access. Enabling delivery requires Admin or Editor access on the source project. Organization administration logs are described as upcoming, not part of this release.

### Source excerpt

Eight alerting rules and seven dashboards arrive with the logs, including Kibana Object Forensics, where a deleted detection rule shows up with the person who deleted it and the space it was in.

## Evaluate AI models for observability: o11y-bench joins Fireworks' Specialized Intelligence Index and reaches v1.1

DevFeed: [Evaluate AI models for observability: o11y-bench joins Fireworks' Specialized Intelligence Index and reaches v1.1](<https://devfeed.tech/articles/evaluate-ai-models-for-observability-o11y-bench-joins-fireworks-specialized-intelligence-index-and-reaches-v1-1-105724.md>)

Original publisher: [Read original article](<https://grafana.com/blog/evaluate-ai-models-for-observability-o11y-bench-joins-fireworks-specialized-intelligence-index-and-reaches-v1-1/>)

Author: Sean Porter

Published: 2026-10-28T16:08:24Z

Content type: release

Language: en

Sources: [Grafana Labs](<https://devfeed.tech/sources/grafana-labs-blog-on-grafana-labs.md>)

Topics: [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>), [observability](<https://devfeed.tech/topics/observability.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Grafana](<https://devfeed.tech/topics/grafana.md>), [telemetry](<https://devfeed.tech/topics/telemetry.md>), [litellm](<https://devfeed.tech/topics/litellm.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-models](<https://devfeed.tech/tags/ai-models.md>), [ai-training](<https://devfeed.tech/tags/ai-training.md>), [ai-training-and-inference](<https://devfeed.tech/tags/ai-training-and-inference.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [bench](<https://devfeed.tech/tags/bench.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [dashboards](<https://devfeed.tech/tags/dashboards.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [grafana](<https://devfeed.tech/tags/grafana.md>), [grafana-labs](<https://devfeed.tech/tags/grafana-labs.md>), [index](<https://devfeed.tech/tags/index.md>), [inference](<https://devfeed.tech/tags/inference.md>), [integration](<https://devfeed.tech/tags/integration.md>), [intelligence](<https://devfeed.tech/tags/intelligence.md>), [litellm](<https://devfeed.tech/tags/litellm.md>), [logs](<https://devfeed.tech/tags/logs.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [models](<https://devfeed.tech/tags/models.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [observability](<https://devfeed.tech/tags/observability.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [traces](<https://devfeed.tech/tags/traces.md>)

### AI overview

o11y-bench evaluates whether AI agents can complete observability and SRE tasks using evidence from a running Grafana stack, rather than merely produce plausible answers. Agents use the Grafana MCP server on synthetic telemetry across 64 tasks in six areas; each task runs three times, with Pass³ measuring average performance and Pass@3 recording whether the agent succeeds at least once. Version 1.1 checks claims against environment facts, verifies cited traces in Tempo, grades dashboard changes through saved-state diffs, and treats failed ground-truth queries as grading errors. Saved facts allow regrading without rerunning agents. A no-tools control failed all 64 tasks. The benchmark is open source and now supports Fireworks inference for serverless and account-deployed models through its LiteLLM-backed agent. Version 1.1 results establish a new baseline and are not directly comparable with 1.0.

### Source excerpt

When evaluating AI models for observability, generating a plausible query or explanation isn't enough. Teams need to know whether an agent's conclusions are supported by telemetry, whether its changes to dashboards actually work, and whether it can deliver reliable results consistently. This is why we built o11y-bench, Grafana Labs' open benchmark for evaluating AI models on observability and SRE tasks. And today, we're excited to share two updates: o11y-bench is now part of the Specialized Intelligence Index from Fireworks, the AI training and inference platform, and we're releasing o11y-bench 1.1, with updates that make grading more grounded, consistent, and reproducible. Bringing o11y-bench to the Specialized Intelligence Index The Specialized Intelligence Index brings together practitioner-built benchmarks for comparing how open, closed, and specialized AI models perform on domain-specific work, from healthcare and finance to cybersecurity and software engineering. It helps teams identify the best models for their workloads, understand where specialization adds value, and evaluate tradeoffs between quality, cost, and speed. With o11y-bench now part of the index, SRE, platform engineering, and observability teams can explore how AI models perform on real observability workflows--from querying metrics, logs, and traces to investigating incidents and managing Grafana dashboards--alongside benchmarks for other specialized domains and use cases. Run the benchmark with Fireworks inference o11y-bench also now supports Fireworks as an inference provider in its default LiteLLM-backed agent. This means teams can evaluate both serverless models and models deployed within their own Fireworks accounts, using the same reproducible workflow available for other providers. The integration handles Fireworks credentials, deployment names, and reasoning settings so benchmark runs are configured consistently. This makes it easier to reproduce results from the index or evaluate Firewor

## Your AI agents have audit logs. Can you reconstruct their decisions?

DevFeed: [Your AI agents have audit logs. Can you reconstruct their decisions?](<https://devfeed.tech/articles/your-ai-agents-have-audit-logs-can-you-reconstruct-their-decisions-104242.md>)

Original publisher: [Read original article](<https://www.cio.com/article/4241123/your-ai-agents-have-audit-logs-can-you-reconstruct-their-decisions.html>)

Author: Maman Ibrahim

Published: 2026-10-28T09:00:00Z

Content type: article

Language: en

Sources: [CIO](<https://devfeed.tech/sources/cio.md>)

Topics: [ai observability](<https://devfeed.tech/topics/ai-observability.md>), [audit](<https://devfeed.tech/topics/audit.md>), [telemetry](<https://devfeed.tech/topics/telemetry.md>), [agent orchestration](<https://devfeed.tech/topics/agent-orchestration.md>), [ML model versioning](<https://devfeed.tech/topics/ml-model-versioning.md>), [Threat monitoring](<https://devfeed.tech/topics/threat-monitoring.md>)

Tags: [accountability](<https://devfeed.tech/tags/accountability.md>), [actions](<https://devfeed.tech/tags/actions.md>), [agents](<https://devfeed.tech/tags/agents.md>), [agents-in-production](<https://devfeed.tech/tags/agents-in-production.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [artificial-intelligence-it-governance-it-leadership](<https://devfeed.tech/tags/artificial-intelligence-it-governance-it-leadership.md>), [audit](<https://devfeed.tech/tags/audit.md>), [audit-logs](<https://devfeed.tech/tags/audit-logs.md>), [board](<https://devfeed.tech/tags/board.md>), [contributor](<https://devfeed.tech/tags/contributor.md>), [edition-us](<https://devfeed.tech/tags/edition-us.md>), [it-governance](<https://devfeed.tech/tags/it-governance.md>), [it-leadership](<https://devfeed.tech/tags/it-leadership.md>), [language-en](<https://devfeed.tech/tags/language-en.md>), [logging](<https://devfeed.tech/tags/logging.md>), [logs](<https://devfeed.tech/tags/logs.md>), [models](<https://devfeed.tech/tags/models.md>), [policies](<https://devfeed.tech/tags/policies.md>), [records](<https://devfeed.tech/tags/records.md>), [storytype-opinion](<https://devfeed.tech/tags/storytype-opinion.md>), [team](<https://devfeed.tech/tags/team.md>), [tool](<https://devfeed.tech/tags/tool.md>), [tooling](<https://devfeed.tech/tags/tooling.md>)

### AI overview

Ordinary event logs often cannot explain an AI agent's decision because outputs vary, one decision spans multiple steps, models and configurations change, and authority is split across systems. The article frames reconstruction as a records and governance problem, distinct from model interpretability, and identifies four elements to preserve: the exact information available at the time, the rule or goal behind the choice, the delegation and accountable human, and the action's downstream effects. It recommends testing current records with a reconstruction drill, scaling capture to a decision's impact and reversibility, and protecting records with suitable retention and tamper-evident storage separate from the systems that create them. The EU AI Act is cited as requiring traceable logs for high-risk systems and explanations for certain significant decisions.

### Source excerpt

The moment arrives after the damage. An agent has cancelled a supplier contract, approved a refund it should have declined or moved a customer record somewhere it should never have gone. Someone senior asks a simple question. Why did it do that? The engineering team opens the logs. There are thousands of lines. Every call is timestamped. Every tool invocation is recorded. And nobody in the room can say why the agent chose what it chose. This pattern is becoming familiar. It will become more so. Logging in most enterprises is mature. Security teams have spent two decades building collection pipelines, retention policies and search tooling. The natural assumption is that this investment transfers to AI agents. It does not. Logs capture events. Accountability requires decisions that can be rebuilt after the fact and rebuilding a decision needs four elements that most logging captures only in part. Why logs fall short for agents An event log records that something happened and when. A decision record captures the basis for that decision. People often confuse the two because, in conventional software, the distinction rarely mattered. Code was deterministic. Anyone who knew the inputs and the version could rerun the logic and arrive at the same result. The log was enough because you could interrogate the system itself. Agents break this assumption in four ways. First, their output varies. Running the same scenario twice can produce two different answers. Replay, the standard investigative move, no longer works. Second, a single decision spans many steps. An agent that resolves a customer complaint might retrieve a policy, query an account, call a pricing tool, consult a second agent and then act. The log shows twelve entries. The decision was one. Third, the system that made the decision may no longer exist. Models are updated. Prompts are revised. Tool configurations change weekly. Six months later, the agent that made the call has been replaced by something that behaves

## Annotrace: A Workbench for Step-Level Annotation of Agent Traces

DevFeed: [Annotrace: A Workbench for Step-Level Annotation of Agent Traces](<https://devfeed.tech/articles/annotrace-a-workbench-for-step-level-annotation-of-agent-traces-103024.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.23284>)

Author: Chenyi Huang, Arun Josephraj Arokiaraj, Bin Wu, Sahan Bulathwela

Published: 2026-10-28T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [agent observability](<https://devfeed.tech/topics/agent-observability.md>), [ai observability](<https://devfeed.tech/topics/ai-observability.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [telemetry](<https://devfeed.tech/topics/telemetry.md>), [Tool](<https://devfeed.tech/topics/tool.md>), [Multi Agent Systems](<https://devfeed.tech/topics/multi-agent-systems.md>), [OpenTelemetry](<https://devfeed.tech/topics/opentelemetry.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [annotation](<https://devfeed.tech/tags/annotation.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-hc](<https://devfeed.tech/tags/cs-hc.md>), [cs-ma](<https://devfeed.tech/tags/cs-ma.md>), [frontier-model](<https://devfeed.tech/tags/frontier-model.md>), [genai](<https://devfeed.tech/tags/genai.md>), [langfuse](<https://devfeed.tech/tags/langfuse.md>), [langsmith](<https://devfeed.tech/tags/langsmith.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-agents](<https://devfeed.tech/tags/llm-agents.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [multi-agent](<https://devfeed.tech/tags/multi-agent.md>), [nested](<https://devfeed.tech/tags/nested.md>), [observability](<https://devfeed.tech/tags/observability.md>), [opentelemetry](<https://devfeed.tech/tags/opentelemetry.md>), [opentelemetry-genai](<https://devfeed.tech/tags/opentelemetry-genai.md>), [platforms](<https://devfeed.tech/tags/platforms.md>), [traces](<https://devfeed.tech/tags/traces.md>)

### AI overview

Annotrace structures agent runs from six trace formats, including OpenTelemetry GenAI, LangSmith, and Langfuse, as ordered, nested steps for annotation. Annotators supply a codebook; a frontier model proposes an initial set from sampled traces, labels first, and answers the same questions as the human, enabling human-AI annotation and agreement measurement between any two annotators. On a multi-agent question-answering corpus, it reproduced 235 annotation units from prior manual work exactly; two models labeling the same traces agreed 91% of the time (κ = 0.82).

### Source excerpt

arXiv:2610.23284v3 Announce Type: replace Abstract: Every study of why LLM agents fail begins by labelling traces by hand, and every such study builds its own throwaway tool. Observability platforms record agent traces but offer no structured labelling; annotation tools label text but have no notion of a step, a delegation or a tool call, leaving it unclear what the smallest unit of execution to label should be. Annotrace reads six trace formats, including OpenTelemetry GenAI, LangSmith and Langfuse, and turns each run into an ordered, nested list of steps. The annotator writes their own labelling codebook; a frontier model proposes a starting set from a sample of their own traces, answers first, and is asked exactly the questions the human is asked, making annotation a joint human-AI process. Annotrace measures agreement between any two annotators, human or AI. On a multi-agent question-answering corpus it reproduces a previous manual effort's 235 annotation units exactly, and two models labelling the same traces agree 91% of the time at $\kappa = 0.82$.

## Too many clients spoil the logs: distributed parsing, aggregation techniques and metrics

DevFeed: [Too many clients spoil the logs: distributed parsing, aggregation techniques and metrics](<https://devfeed.tech/articles/too-many-clients-spoil-the-logs-distributed-parsing-aggregation-techniques-and-metrics-103371.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.36946>)

Author: Andr\\'e Garc\\'ia G\\'omez, Viktor Beck, Max Landauer, Markus Wurzenberger, Florian Skopik, Andreas Rauber, Edgar Weipp

Published: 2026-10-28T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [Tree-sitter](<https://devfeed.tech/topics/tree-sitter.md>), [Logging and metrics](<https://devfeed.tech/topics/logging-and-metrics.md>), [telemetry](<https://devfeed.tech/topics/telemetry.md>)

Tags: [aggregation](<https://devfeed.tech/tags/aggregation.md>), [algorithms](<https://devfeed.tech/tags/algorithms.md>), [anomaly-detection](<https://devfeed.tech/tags/anomaly-detection.md>), [clients](<https://devfeed.tech/tags/clients.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [federated](<https://devfeed.tech/tags/federated.md>), [iot](<https://devfeed.tech/tags/iot.md>), [logs](<https://devfeed.tech/tags/logs.md>), [mappings](<https://devfeed.tech/tags/mappings.md>), [messages](<https://devfeed.tech/tags/messages.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [parsing](<https://devfeed.tech/tags/parsing.md>), [performance](<https://devfeed.tech/tags/performance.md>)

### AI overview

Federated log anomaly detection depends on parsers to extract templates at inference time, but deploying parsers across organizations remains an open challenge. The study empirically investigates distributed parsing, evaluates five parsing algorithms with aggregation techniques for reaching unified event-template mappings across clients, and tests 6,720 distributed configurations. It also introduces metrics intended to give a more comprehensive view of parser behavior in distributed environments and reports lessons and recommendations for design, deployment, and evaluation. The supplied text does not state which algorithms or configurations performed best.

### Source excerpt

arXiv:2610.36946v1 Announce Type: new Abstract: Log messages are employed to identify anomalies in IoT systems and have been widely used across many applications. To enable collaboration among infrastructures belonging to different organizations, federated log anomaly detection has been thoroughly investigated for this purpose, often assuming the availability of ground truth log templates. However, this is typically not the case, as it is still necessary to depend on log parsers to extract log templates at inference time, while the design and deployment of such parsers in distributed environments remains largely an open problem. To this end, we report on an empirical investigation of distributed parsing and its key challenges, providing insights and lessons learned towards closing the gap between research and application. To further refine and contextualize the resulting benchmarks, we examine and introduce new metrics that provide a more comprehensive view for evaluating distributed parsers and how they behave in distributed environments. We propose a set of lessons learned and recommendations to support the design, deployment, and evaluation of distributed parsers. In particular, we evaluate the performance of five distinct parsing algorithms in combination with aggregation techniques that allow a unified agreement of event-template mappings across all participating clients, a prerequisite for federated log anomaly detection. In total, 6,720 distinct distributed configurations were tested for this purpose.

[Next page](<https://devfeed.tech/topics/telemetry.md?cursor=WyIyMDI2LTEwLTI4VDA0OjAwOjAwKzAwOjAwIiwgIjY0ZTRiNjY4LTZjZjMtNDU4Mi05MTRiLThlMDQ2ZjhlMjU5MCJd>)