# ai observability

The practice of monitoring, understanding, troubleshooting, and evaluating AI systems through their lifecycle using signals such as traces, metrics, logs, and model outputs.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Datadog LLM Observability: Features, Pricing, and Alternatives

DevFeed: [Datadog LLM Observability: Features, Pricing, and Alternatives](<https://devfeed.tech/articles/datadog-llm-observability-features-pricing-and-alternatives-145962.md>)

Original publisher: [Read original article](<https://newrelic.com/blog/ai/datadog-llm-observability>)

Author: John Blust

Published: 2026-11-20T11:51:52.599709Z

Content type: article

Language: en

Sources: [New Relic](<https://devfeed.tech/sources/new-relic.md>)

Topics: [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [datadog](<https://devfeed.tech/topics/datadog.md>), [agent observability](<https://devfeed.tech/topics/agent-observability.md>), [ai observability](<https://devfeed.tech/topics/ai-observability.md>), [observability](<https://devfeed.tech/topics/observability.md>), [tracing](<https://devfeed.tech/topics/tracing.md>), [OpenTelemetry](<https://devfeed.tech/topics/opentelemetry.md>), [LLM security](<https://devfeed.tech/topics/llm-security.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [phoenix](<https://devfeed.tech/topics/phoenix.md>), [LangChain](<https://devfeed.tech/topics/langchain.md>), [dynatrace](<https://devfeed.tech/topics/dynatrace.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-monitoring](<https://devfeed.tech/tags/ai-monitoring.md>), [alternatives](<https://devfeed.tech/tags/alternatives.md>), [compare](<https://devfeed.tech/tags/compare.md>), [datadog](<https://devfeed.tech/tags/datadog.md>), [devops](<https://devfeed.tech/tags/devops.md>), [evaluations](<https://devfeed.tech/tags/evaluations.md>), [features](<https://devfeed.tech/tags/features.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-observability](<https://devfeed.tech/tags/llm-observability.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [new-relic](<https://devfeed.tech/tags/new-relic.md>), [observability](<https://devfeed.tech/tags/observability.md>), [pricing](<https://devfeed.tech/tags/pricing.md>), [tracing](<https://devfeed.tech/tags/tracing.md>)

### AI overview

Datadog LLM Observability, now called Agent Observability, traces model calls and agent workflows, evaluates quality and safety, supports prompt experiments, and tracks latency, tokens, errors, and costs. It is a standalone product billed by model-call spans; tool, retrieval, and workflow spans do not count. Both listed plans include core tracing and evaluation features, and annual Pro pricing starts at $160 per month, with retention add-ons affecting per-span rates. Alternatives differ in approach: New Relic connects AI telemetry to existing APM, including optional eBPF monitoring; Langfuse offers a self-hostable core; LangSmith emphasizes agent tracing and evaluation workflows; and Arize Phoenix supports local deployment with OpenTelemetry-based instrumentation. The text notes that Datadog can correlate with its other products through shared trace IDs and supports Python, Node.js, Java, OpenTelemetry, and HTTP API integration.

### Source excerpt

An objective look at Datadog LLM Observability: tracing, evaluations, how pricing works, and how alternatives like New Relic AI Monitoring compare.

## Evaluating Governed LLM-Driven ML Experimentation: Lifecycle, Provenance, and Failure Metrics

DevFeed: [Evaluating Governed LLM-Driven ML Experimentation: Lifecycle, Provenance, and Failure Metrics](<https://devfeed.tech/articles/evaluating-governed-llm-driven-ml-experimentation-lifecycle-provenance-and-failure-metrics-142192.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23678>)

Author: Dowling Wong

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [ai observability](<https://devfeed.tech/topics/ai-observability.md>), [agentic workflows](<https://devfeed.tech/topics/agentic-workflows.md>), [audit](<https://devfeed.tech/topics/audit.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Open Data](<https://devfeed.tech/topics/open-data.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [experimentation](<https://devfeed.tech/tags/experimentation.md>), [failure](<https://devfeed.tech/tags/failure.md>), [governance](<https://devfeed.tech/tags/governance.md>), [harness](<https://devfeed.tech/tags/harness.md>), [ledger](<https://devfeed.tech/tags/ledger.md>), [lifecycle](<https://devfeed.tech/tags/lifecycle.md>), [llm](<https://devfeed.tech/tags/llm.md>), [memory](<https://devfeed.tech/tags/memory.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [ml](<https://devfeed.tech/tags/ml.md>), [process](<https://devfeed.tech/tags/process.md>), [provenance](<https://devfeed.tech/tags/provenance.md>)

### AI overview

The study evaluates LLM-driven ML experimentation by process as well as best score. Its governed harness assigns trial lifecycle, edit-scope checks, keep-or-discard decisions, and append-only logging to a control plane rather than the proposing LLM; governance metrics are computed from that ledger. Across six tasks and 1,445 trials, the same contract exposed failures hidden by task scores, including 120 failed trials in a no-memory arm and a reset failure detected by reproducibility hashes. In a matched comparison, success-only logging omitted 43 of 50 trials, while the governed ledger recorded all 50. Memory's effect on repeated bad proposals varied by task. The authors say they release all trial ledgers so reported numbers can be recomputed.

### Source excerpt

arXiv:2611.23678v1 Announce Type: cross Abstract: Autonomous agents that run ML experiments are usually judged by the best score they reach. That score says nothing about how it was reached: whether the agent stayed within the files it was allowed to edit, whether failed trials were kept or silently dropped, and whether each keep or discard decision can be traced to its evidence. We propose evaluating such systems on their experimental process as well as their outcome, and we build a governed harness that makes this possible. A control plane, not the LLM that proposes changes, owns each trial's lifecycle, checks its edit scope, decides keep or discard, and appends one record per trial to an append-only ledger. Governance metrics such as the invalid-action rate, the repeated-bad rate and provenance completeness are computed from that ledger. Across six ML tasks, from synthetic logistic regression to language-model training, and 1,445 trials, the same contract applied unchanged. The ledger exposed failures that a task score hides: a no-memory arm in which all 120 trials failed, a benchmark adapter whose bounded proposals ran out, and a reset failure that reproducibility hashes revealed, after which the affected campaigns were rerun. In a matched counterfactual, a loop that logs only successful trials left no record of 43 of 50 trials, while the governed ledger recorded all 50. Memory ablations show that memory's effect on repeated bad proposals depends on the task. We release the ledgers of all 1,445 trials, so that every number reported for them can be recomputed.

## Can LLMs Understand What They Predict? Evaluating Latent Causal Structure Inference in Financial Markets

DevFeed: [Can LLMs Understand What They Predict? Evaluating Latent Causal Structure Inference in Financial Markets](<https://devfeed.tech/articles/can-llms-understand-what-they-predict-evaluating-latent-causal-structure-inference-in-financial-markets-143404.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23533>)

Author: Yixiang Zheng, Yuechen Jiang, Zhiwei Liu, Jimin Huang, Yuyan Wang, Sophia Ananiadou

Published: 2026-11-19T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Predictive AI](<https://devfeed.tech/topics/predictive-ai.md>), [ai observability](<https://devfeed.tech/topics/ai-observability.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [event](<https://devfeed.tech/tags/event.md>), [financial](<https://devfeed.tech/tags/financial.md>), [financial-markets](<https://devfeed.tech/tags/financial-markets.md>), [graphs](<https://devfeed.tech/tags/graphs.md>), [inference](<https://devfeed.tech/tags/inference.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [llms](<https://devfeed.tech/tags/llms.md>)

### AI overview

LUCID tests whether LLMs can infer latent market structures from event descriptions and historical financial observations, rather than merely predict outcomes. It evaluates model-generated event-induced causal graphs through reconstruction. Across six open-weight and proprietary LLMs, adding the generated graphs generally reduced reconstruction quality compared with using historical observations alone, though a few metrics improved. The results indicate that the evaluated models struggle to infer informative, event-specific market structures from observable evidence.

### Source excerpt

arXiv:2611.23533v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly evaluated through their ability to predict future outcomes, implicitly assuming that predictive success reflects an understanding of the underlying system. However, accurate predictions can often be achieved without recovering the latent mechanisms that generate observable outcomes. To examine this assumption, we introduce LUCID, a benchmark that evaluates whether models can infer latent market structures from event descriptions and historical financial observations. Models generate event-induced causal graphs representing inferred market structure and are evaluated through a reconstruction-based framework. Our evaluation of 6 open-weight and proprietary LLMs shows that adding their generated graphs generally reduces reconstruction quality compared with using historical observations alone, although a few metrics improve. These findings suggest that inferring informative, event-specific market structures from observable evidence remains challenging for the evaluated LLMs.

## MANGO Arena: Same Seeds, Different Chains Evaluating LLM Agents in a Disrupted Supply Chain

DevFeed: [MANGO Arena: Same Seeds, Different Chains Evaluating LLM Agents in a Disrupted Supply Chain](<https://devfeed.tech/articles/mango-arena-same-seeds-different-chains-evaluating-llm-agents-in-a-disrupted-supply-chain-142680.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23613>)

Author: Sara Amar

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Multi Agent Systems](<https://devfeed.tech/topics/multi-agent-systems.md>), [reproducibility](<https://devfeed.tech/topics/reproducibility.md>), [ai observability](<https://devfeed.tech/topics/ai-observability.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [api](<https://devfeed.tech/tags/api.md>), [arena](<https://devfeed.tech/tags/arena.md>), [batches](<https://devfeed.tech/tags/batches.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [constraint](<https://devfeed.tech/tags/constraint.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-ma](<https://devfeed.tech/tags/cs-ma.md>), [deal](<https://devfeed.tech/tags/deal.md>), [demand](<https://devfeed.tech/tags/demand.md>), [disrupted](<https://devfeed.tech/tags/disrupted.md>), [disruption](<https://devfeed.tech/tags/disruption.md>), [environment](<https://devfeed.tech/tags/environment.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [evaluations](<https://devfeed.tech/tags/evaluations.md>), [factory](<https://devfeed.tech/tags/factory.md>), [failed](<https://devfeed.tech/tags/failed.md>), [failures](<https://devfeed.tech/tags/failures.md>), [fire](<https://devfeed.tech/tags/fire.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-agents](<https://devfeed.tech/tags/llm-agents.md>), [supply-chain](<https://devfeed.tech/tags/supply-chain.md>)

### AI overview

MANGO Arena evaluates four LLM agents managing a serial supply chain under truck, cash and production constraints, while tracking bullwhip, rule adherence and collusion proposals. In a pilot with three small reasoning models, all amplified demand variability upstream: 7-25x at the Retailer and 13-179x at the Supplier, often through batch ordering. Rule violations were rare and no agent proposed a money side deal. The results were highly sensitive to evaluation reliability: API failures silently counted as no orders, and upstream bullwhip varied by up to 17x across reruns with the same seeds and no failed calls. With only three seeds, conclusions about disruption effects did not hold on rerun. The authors report these failures and propose practices for multi-agent evaluations.

### Source excerpt

arXiv:2611.23613v1 Announce Type: cross Abstract: We introduce MANGO Arena, an open-source testbed in which four LLM agents each run one company in a serial supply chain adapted from MANGO, a classroom factory-management game. The environment enforces hard rules (a truck limit, a no-credit cash constraint and production capacity), logs every blocked attempt to break them, and gives agents a public message board on which side payments are explicitly forbidden. This lets one environment measure three things at once: an economic outcome (the bullwhip effect), rule adherence under pressure, and proposals to collude. In a pilot with three small reasoning models, two scripted reference policies, two conditions (with and without a supplier fire) and three demand seeds, all LLM agents amplified demand variability as it moved upstream, by 7 to 25 times at the Retailer and 13 to 179 times at the Supplier, often more than a textbook order-up-to policy. They did so largely by ordering in batches (zero orders in up to 60% of decisions). Real rule violations were rare (at most 7 of 288 decisions per model) and no agent proposed a money side deal. The pilot's main lesson concerns evaluation method. API failures (rate limits and runaway hidden reasoning) entered the metrics silently as "no order" decisions. And when we reran games with the same seeds and no failed calls, upstream bullwhip changed by up to 17 times between runs. With three seeds, conclusions about how a disruption affects a given model did not survive a rerun. We report these failures in full and propose reporting practices for multi-agent evaluations of this kind. Code, prompts and every logged decision are public.

## Tracing the Messy Middle: A Runtime Transaction Schema for AI Coding Agents

DevFeed: [Tracing the Messy Middle: A Runtime Transaction Schema for AI Coding Agents](<https://devfeed.tech/articles/tracing-the-messy-middle-a-runtime-transaction-schema-for-ai-coding-agents-142123.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23234>)

Author: Yuelin Hou, Haorui Li, Lu Chen, Zhen Chen

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [ai observability](<https://devfeed.tech/topics/ai-observability.md>), [tracing](<https://devfeed.tech/topics/tracing.md>), [schema design](<https://devfeed.tech/topics/schema-design.md>)

Tags: [ai-coding-agents](<https://devfeed.tech/tags/ai-coding-agents.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [atm](<https://devfeed.tech/tags/atm.md>), [causality](<https://devfeed.tech/tags/causality.md>), [coaching](<https://devfeed.tech/tags/coaching.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [fallback](<https://devfeed.tech/tags/fallback.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [local-first](<https://devfeed.tech/tags/local-first.md>), [observability](<https://devfeed.tech/tags/observability.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [productivity](<https://devfeed.tech/tags/productivity.md>), [reconciliation](<https://devfeed.tech/tags/reconciliation.md>), [retries](<https://devfeed.tech/tags/retries.md>), [routing-policy](<https://devfeed.tech/tags/routing-policy.md>), [runtime](<https://devfeed.tech/tags/runtime.md>), [schema](<https://devfeed.tech/tags/schema.md>), [sessions](<https://devfeed.tech/tags/sessions.md>), [tracing](<https://devfeed.tech/tags/tracing.md>), [transaction](<https://devfeed.tech/tags/transaction.md>)

### AI overview

The paper proposes recording AI coding agents' runtime activity as structured transactions to expose retries, fallbacks, routing choices, and abandoned sessions that final artifacts miss. Such records could support behavior-based engineering coaching, workflow improvements at failure points, and routing decisions based on total task cost rather than per-request price. The authors describe a runtime transaction schema and initial feasibility with Agent Token Manager (Atm), a local-first prototype that reconciles gateway records with transcript evidence without sending prompt content to external collectors. Scaling this approach still faces challenges: reconciling traces without shared request IDs, linking outcomes across asynchronous systems, and establishing whether interventions cause improvements.

### Source excerpt

arXiv:2611.23234v1 Announce Type: new Abstract: AI coding agents are increasingly integrated into software development, yet the field lacks a systematic feedback loop to improve their behavior in practice. Artifact datasets capture final outputs such as code patches, commits, and benchmark outcomes, but omit the messy middle of the development lifecycle: the failed retries, fallback strategies, routing decisions, and abandoned sessions that determine operational cost, reliability, and productivity. We propose an observability framework that captures these runtime behaviors as structured transaction records, enabling engineering coaching grounded in behavioral evidence, workflow optimization at the stages where tasks break down, and routing policy iteration measured by total task cost rather than per-request price. We define the runtime transaction schema this architecture requires and demonstrate initial feasibility through Agent Token Manager (Atm), a local-first prototype that reconciles gateway records and transcript evidence without exposing prompt content to external collectors. Finally, we identify three open research problems that must be solved to scale these feedback mechanisms: trace reconciliation without shared request identifiers, outcome linking across asynchronous systems, and intervention causality. Treating the linkage between sessions and artifacts as a fundamental infrastructure requirement provides a structured foundation for understanding and optimizing agent behavior.

## Log Analysis for LLM-Based Agents: A Lifecycle-Oriented Survey

DevFeed: [Log Analysis for LLM-Based Agents: A Lifecycle-Oriented Survey](<https://devfeed.tech/articles/log-analysis-for-llm-based-agents-a-lifecycle-oriented-survey-142150.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23281>)

Author: Cong Wang, Yaowei Sun, Guohua Xin, Jiongchi Yu, Shuangcheng Sang, Guangquan Xu, Maode Ma

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [ai observability](<https://devfeed.tech/topics/ai-observability.md>), [log management](<https://devfeed.tech/topics/log-management.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [based](<https://devfeed.tech/tags/based.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [lifecycle](<https://devfeed.tech/tags/lifecycle.md>), [llm](<https://devfeed.tech/tags/llm.md>), [log](<https://devfeed.tech/tags/log.md>), [logging](<https://devfeed.tech/tags/logging.md>), [logs](<https://devfeed.tech/tags/logs.md>), [mining](<https://devfeed.tech/tags/mining.md>), [survey](<https://devfeed.tech/tags/survey.md>)

### AI overview

A lifecycle-oriented review of 90 studies published from 2024 through August 2026 examines LLM-agent log analysis across logging, structuring, and mining. It identifies dependencies and limitations between stages: detailed logs may still fail to provide sufficient evidence, and uniform structure does not ensure semantic equivalence. The authors argue analytical claims should align with supporting evidence, reference criteria, and validation procedures. The abstract also says the survey summarizes representative open-source tools, public datasets, and research directions, but does not name them.

### Source excerpt

arXiv:2611.23281v1 Announce Type: new Abstract: Logs generated by LLM-based agents support analysis of execution behavior beyond final outputs. Existing surveys give limited attention to dependencies across the lifecycle of these records. A lifecycle-oriented review of 90 core studies published between 2024 and August 2026 covers Agent Logging, Agent Log Structuring, and Agent Log Mining. Stage-specific comparisons reveal cross-stage dependencies and limitations. Detailed logs do not necessarily provide sufficient evidence. Structural uniformity does not guarantee semantic equivalence. Analytical claims require alignment with supporting evidence, reference criteria, and validation procedures. Representative open-source tools, public datasets, and future research directions are also summarized.

## What Is LLM Monitoring? What to Track Once Your Model Is Live

DevFeed: [What Is LLM Monitoring? What to Track Once Your Model Is Live](<https://devfeed.tech/articles/what-is-llm-monitoring-what-to-track-once-your-model-is-live-141933.md>)

Original publisher: [Read original article](<https://montecarlo.ai/blog-llm-monitoring>)

Author: Ruslan Sultanov

Published: 2026-11-18T18:07:29Z

Content type: article

Language: en

Sources: [Monte Carlo](<https://devfeed.tech/sources/monte-carlo.md>)

Topics: [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [Monitoring](<https://devfeed.tech/topics/monitoring.md>), [ai observability](<https://devfeed.tech/topics/ai-observability.md>), [Monitoring & Alerting](<https://devfeed.tech/topics/monitoring-alerting.md>), [data observability](<https://devfeed.tech/topics/data-observability.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [agent-trust](<https://devfeed.tech/tags/agent-trust.md>), [agent-trust-glossary](<https://devfeed.tech/tags/agent-trust-glossary.md>), [glossary](<https://devfeed.tech/tags/glossary.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-monitoring](<https://devfeed.tech/tags/llm-monitoring.md>), [llm-monitoring-metrics](<https://devfeed.tech/tags/llm-monitoring-metrics.md>), [llm-monitoring-vs-observability](<https://devfeed.tech/tags/llm-monitoring-vs-observability.md>), [llm-performance-monitoring](<https://devfeed.tech/tags/llm-performance-monitoring.md>), [llm-quality-monitoring](<https://devfeed.tech/tags/llm-quality-monitoring.md>), [model](<https://devfeed.tech/tags/model.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [monitoring-llms-in-production](<https://devfeed.tech/tags/monitoring-llms-in-production.md>), [what-is-llm-monitoring](<https://devfeed.tech/tags/what-is-llm-monitoring.md>)

### AI overview

LLM monitoring catches production failures that uptime, latency, and error dashboards miss: an app can be fast and available while returning incorrect or unsafe answers. It continuously tracks performance (latency, tokens, cost, errors), context (source freshness and retrieval), behavior (tool calls, retries, refusals), and output quality (groundedness, relevance, correctness, safety, and user feedback). A practical setup traces requests across retrieval, model, and tool calls; scores traffic with deterministic checks and LLM-as-judge evaluations; alerts on trends or anomalies; then uses trace context to investigate causes such as stale data, prompt changes, or model updates. Evals assess fixed examples before release, monitoring detects production changes, and observability supplies traces and lineage to explain them. The article recommends versioning traces, establishing baselines, checking judges against human review, and assigning alert owners.

### Source excerpt

Your LLM app has been live for a month. Latency is under a second, the error rate is zero, and uptime is perfect. Meanwhile, a product manager forwards a screenshot: the bot told a customer about a discount that ended in March. Nothing on the dashboard moved. That's the gap LLM monitoring fills. This guide ...

## What Is Agent Tracing? How to See Every Step an Agent Takes

DevFeed: [What Is Agent Tracing? How to See Every Step an Agent Takes](<https://devfeed.tech/articles/what-is-agent-tracing-how-to-see-every-step-an-agent-takes-141927.md>)

Original publisher: [Read original article](<https://montecarlo.ai/blog-agent-tracing>)

Author: Ruslan Sultanov

Published: 2026-11-18T17:20:39Z

Content type: article

Language: en

Sources: [Monte Carlo](<https://devfeed.tech/sources/monte-carlo.md>)

Topics: [ai observability](<https://devfeed.tech/topics/ai-observability.md>), [tracing](<https://devfeed.tech/topics/tracing.md>), [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [Multi-step AI processes](<https://devfeed.tech/topics/multi-step-ai-processes.md>), [Latency](<https://devfeed.tech/topics/latency.md>)

Tags: [accountability](<https://devfeed.tech/tags/accountability.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agent-spans](<https://devfeed.tech/tags/agent-spans.md>), [agent-tracing](<https://devfeed.tech/tags/agent-tracing.md>), [agent-tracing-vs-logging](<https://devfeed.tech/tags/agent-tracing-vs-logging.md>), [agent-trust](<https://devfeed.tech/tags/agent-trust.md>), [agent-trust-glossary](<https://devfeed.tech/tags/agent-trust-glossary.md>), [ai-agent-tracing](<https://devfeed.tech/tags/ai-agent-tracing.md>), [black-box](<https://devfeed.tech/tags/black-box.md>), [customer](<https://devfeed.tech/tags/customer.md>), [glossary](<https://devfeed.tech/tags/glossary.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [latency](<https://devfeed.tech/tags/latency.md>), [llm-tracing](<https://devfeed.tech/tags/llm-tracing.md>), [logs](<https://devfeed.tech/tags/logs.md>), [model](<https://devfeed.tech/tags/model.md>), [opentelemetry-genai-semantic-conventions](<https://devfeed.tech/tags/opentelemetry-genai-semantic-conventions.md>), [order](<https://devfeed.tech/tags/order.md>), [request](<https://devfeed.tech/tags/request.md>), [response](<https://devfeed.tech/tags/response.md>), [service](<https://devfeed.tech/tags/service.md>), [tools](<https://devfeed.tech/tags/tools.md>), [trace-ai-agents](<https://devfeed.tech/tags/trace-ai-agents.md>), [tracing](<https://devfeed.tech/tags/tracing.md>), [what-is](<https://devfeed.tech/tags/what-is.md>), [what-is-agent-tracing](<https://devfeed.tech/tags/what-is-agent-tracing.md>)

### AI overview

Agent tracing records an AI agent's full run as linked spans, capturing steps such as planning, retrieval, model calls, and tool use alongside inputs, outputs, timing, and metadata. This makes silent failures, retries, quality issues, and cost or latency drivers easier to investigate than ordinary logs or service-focused distributed traces. The excerpt recommends OpenTelemetry GenAI conventions, trace IDs and parent-span IDs, versioning prompts and models, linking retrieved items to source data, and scoring traces. Capture sensitive prompts carefully: redact personal data and set retention policies. OpenTelemetry conventions remain in development, so pin versions. Traces also cannot reveal upstream data freshness problems without lineage or data monitoring. Teams can store traces in their own warehouse, but need tooling to query, score, and alert on them.

### Source excerpt

An agent tells a customer their order shipped. It didn't. Every service returned 200 OK, latency was normal, and the logs show a request in and a response out. What happened in between, which documents the agent read, which tools it called, what the model was told, is a black box. Agent tracing opens that ...

## AI Adoption Maturity Matrix (AAMM): The 5 Levels Every Organization Goes Through

DevFeed: [AI Adoption Maturity Matrix (AAMM): The 5 Levels Every Organization Goes Through](<https://devfeed.tech/articles/ai-adoption-maturity-matrix-aamm-the-5-levels-every-organization-goes-through-140886.md>)

Original publisher: [Read original article](<https://hackernoon.com/ai-adoption-maturity-matrix-aamm-the-5-levels-every-organization-goes-through?source=rss>)

Author: Andre

Published: 2026-11-18T15:48:47Z

Content type: article

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [AI Strategy](<https://devfeed.tech/topics/ai-strategy.md>), [ai evaluation](<https://devfeed.tech/topics/ai-evaluation.md>), [ai observability](<https://devfeed.tech/topics/ai-observability.md>), [decision-making](<https://devfeed.tech/topics/decision-making.md>)

Tags: [adoption](<https://devfeed.tech/tags/adoption.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-adoption](<https://devfeed.tech/tags/ai-adoption.md>), [ai-adoption-maturity-matrix](<https://devfeed.tech/tags/ai-adoption-maturity-matrix.md>), [ai-governance](<https://devfeed.tech/tags/ai-governance.md>), [ai-implementation-strategy](<https://devfeed.tech/tags/ai-implementation-strategy.md>), [ai-maturity](<https://devfeed.tech/tags/ai-maturity.md>), [ai-platform](<https://devfeed.tech/tags/ai-platform.md>), [ai-product-management](<https://devfeed.tech/tags/ai-product-management.md>), [ai-strategy](<https://devfeed.tech/tags/ai-strategy.md>), [ai-transformation](<https://devfeed.tech/tags/ai-transformation.md>), [api](<https://devfeed.tech/tags/api.md>), [assessment](<https://devfeed.tech/tags/assessment.md>), [autonomous](<https://devfeed.tech/tags/autonomous.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [board](<https://devfeed.tech/tags/board.md>), [brand](<https://devfeed.tech/tags/brand.md>), [calendar](<https://devfeed.tech/tags/calendar.md>), [checkpoint](<https://devfeed.tech/tags/checkpoint.md>), [classification](<https://devfeed.tech/tags/classification.md>), [claude](<https://devfeed.tech/tags/claude.md>), [company](<https://devfeed.tech/tags/company.md>), [control](<https://devfeed.tech/tags/control.md>), [enterprise-ai](<https://devfeed.tech/tags/enterprise-ai.md>), [maturity](<https://devfeed.tech/tags/maturity.md>), [organization](<https://devfeed.tech/tags/organization.md>)

### AI overview

The AI Adoption Maturity Matrix (AAMM) assesses AI maturity by workflow changes and measured outcomes, not tool access or successful demos. It describes five ordered levels: experimentation and pilots; workflow redesign that separates human judgment from mechanical work; governance with named owners and logged stop criteria; and measured production operation that uses results to improve the workflow. The text calls workflow redesign the commonly skipped step and says to assess each workflow separately, recording the highest level supported by evidence and the gaps below it. Level 5 retains human controls; it does not mean unsupervised operation. Its reported 96% reduction in human workflow hours applies to one marketing workflow and counts only work handled on the platform, about a third of the benchmark; the remaining time is excluded from the claim.

### Source excerpt

A short self-assessment for leaders who want an honest read on where their AI adoption stands today, and a clear idea of what to build next.

## ARC-2: Continuous Compliance Through Delta Management

DevFeed: [ARC-2: Continuous Compliance Through Delta Management](<https://devfeed.tech/articles/arc-2-continuous-compliance-through-delta-management-139912.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21811>)

Author: Tord Johan Eide, Einar Holt

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [ai regulation](<https://devfeed.tech/topics/ai-regulation.md>), [ai observability](<https://devfeed.tech/topics/ai-observability.md>), [risk-management](<https://devfeed.tech/topics/risk-management.md>)

Tags: [ai-act](<https://devfeed.tech/tags/ai-act.md>), [authorization](<https://devfeed.tech/tags/authorization.md>), [autonomous-systems](<https://devfeed.tech/tags/autonomous-systems.md>), [certification](<https://devfeed.tech/tags/certification.md>), [compliance](<https://devfeed.tech/tags/compliance.md>), [continuous-compliance](<https://devfeed.tech/tags/continuous-compliance.md>), [continuous-monitoring](<https://devfeed.tech/tags/continuous-monitoring.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cy](<https://devfeed.tech/tags/cs-cy.md>), [cs-ro](<https://devfeed.tech/tags/cs-ro.md>), [delta](<https://devfeed.tech/tags/delta.md>), [domain](<https://devfeed.tech/tags/domain.md>), [eu-ai-act](<https://devfeed.tech/tags/eu-ai-act.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [framework](<https://devfeed.tech/tags/framework.md>), [governance](<https://devfeed.tech/tags/governance.md>), [lifecycle](<https://devfeed.tech/tags/lifecycle.md>), [management](<https://devfeed.tech/tags/management.md>), [risk](<https://devfeed.tech/tags/risk.md>), [safety](<https://devfeed.tech/tags/safety.md>), [shutdown](<https://devfeed.tech/tags/shutdown.md>), [trigger](<https://devfeed.tech/tags/trigger.md>), [triggers](<https://devfeed.tech/tags/triggers.md>)

### AI overview

ARC-2 proposes Delta Management to maintain autonomous systems' compliance after deployment by continuously comparing their operational state with a certified baseline region. It extends ARC with four mechanisms: CRAB evaluates the worst verified behavioral deviations while quarantining isolated outliers pending confirmation; a layered trigger taxonomy links safety violations and different forms of degradation to proportionate responses; a graduated safe transition separates withdrawal of operational authorization from physical shutdown; and environmental drift beyond the certified Operational Design Domain prompts recertification or ODD revision rather than automatic authority contraction. The authors present this as a continuous lifecycle governance framework addressing EU AI Act Article 72(2).

### Source excerpt

arXiv:2611.21811v1 Announce Type: cross Abstract: ARC 1 introduced a three-layer governance architecture for autonomous systems comprising model safety validation, cognitive certification (CRAB), and operational authorization (LEA). However, it did not address how compliance should be maintained after deployment. This paper introduces Delta Management, a continuous governance framework that measures deviations between a system's current operational state and its certified baseline, defined as a region rather than a fixed point. Compliance is assessed through continuous monitoring instead of static certification. Four extensions are proposed. First, CRAB evaluation incorporates the worst verified behavioral deviations, with isolated outliers quarantined pending confirmation to prevent unjustified restrictions. Second, a layered trigger taxonomy distinguishes hard safety violations, critical single-dimension failures, correlated multi-system degradation, and risk-weighted composite indicators, each triggering proportionate responses. Third, a graduated safe transition protocol separates withdrawal of operational authorization from physical shutdown, consistent with established safety standards and EU AI Act requirements. Fourth, environmental delta is introduced as a distinct baseline validity condition: environmental drift beyond the certified Operational Design Domain (ODD) triggers recertification or ODD revision rather than automatic authority contraction. Together, these extensions transform ARC from a deployment-time certification architecture into a continuous framework for monitoring, evaluating, and enforcing compliance throughout the operational lifecycle of autonomous systems. The framework addresses the continuous compliance obligations established by EU AI Act Article 72(2).

## How we fine-tuned a faster model for Patterns on Baseten

DevFeed: [How we fine-tuned a faster model for Patterns on Baseten](<https://devfeed.tech/articles/how-we-fine-tuned-a-faster-model-for-patterns-on-baseten-142009.md>)

Original publisher: [Read original article](<https://www.datadoghq.com/blog/fine-tuning-patterns-baseten/>)

Author: Fouad Wahabi; Vincent Cornet; Alex Barksdale

Published: 2026-11-18T00:00:00Z

Content type: article

Language: en

Sources: [Datadog](<https://devfeed.tech/sources/datadog-the-monitor-blog.md>)

Topics: [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [ai observability](<https://devfeed.tech/topics/ai-observability.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Enhancing AI model performance](<https://devfeed.tech/topics/enhancing-ai-model-performance.md>), [baseten](<https://devfeed.tech/topics/baseten.md>), [lora](<https://devfeed.tech/topics/lora.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [model-deployment](<https://devfeed.tech/topics/model-deployment.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [3](<https://devfeed.tech/tags/3.md>), [78](<https://devfeed.tech/tags/78.md>), [about](<https://devfeed.tech/tags/about.md>), [actions](<https://devfeed.tech/tags/actions.md>), [adapter](<https://devfeed.tech/tags/adapter.md>), [after](<https://devfeed.tech/tags/after.md>), [agent-observability](<https://devfeed.tech/tags/agent-observability.md>), [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [baseten](<https://devfeed.tech/tags/baseten.md>), [datadog](<https://devfeed.tech/tags/datadog.md>), [faster](<https://devfeed.tech/tags/faster.md>), [fine](<https://devfeed.tech/tags/fine.md>), [generation](<https://devfeed.tech/tags/generation.md>), [model](<https://devfeed.tech/tags/model.md>), [one](<https://devfeed.tech/tags/one.md>), [patterns](<https://devfeed.tech/tags/patterns.md>), [production](<https://devfeed.tech/tags/production.md>), [time](<https://devfeed.tech/tags/time.md>), [traces](<https://devfeed.tech/tags/traces.md>), [under](<https://devfeed.tech/tags/under.md>), [using](<https://devfeed.tech/tags/using.md>)

### AI overview

Datadog fine-tuned Qwen 3.5 9B for Patterns' repetitive interaction summarization, reducing average generation time on 150 held-out cases from 44.7 seconds to 0.89 seconds. Patterns' production LLM spans supplied examples pairing interactions and clustering instructions with existing summaries; rows were split by trace to limit test leakage. On Baseten, the base model often exhausted its token budget before producing a parseable answer: only 48% of its outputs finished, and 44.7% passed the LLM judge. After LoRA fine-tuning with early stopping based on validation loss, all 150 outputs finished and 93.3% passed. Datadog's pup finetune workflow supported dataset review, benchmarking, training, and evaluation, with approval before paid steps. The reported evaluation used five held-out traces and suggests the model generalized across the tested clustering instructions.

### Source excerpt

Datadog fine-tuned a summarization model using production traces, cutting average generation time from about 45 seconds to under one second.

## Securing Your Agents: A Hands-On Guide to Agent Anomaly Detection

DevFeed: [Securing Your Agents: A Hands-On Guide to Agent Anomaly Detection](<https://devfeed.tech/articles/securing-your-agents-a-hands-on-guide-to-agent-anomaly-detection-136557.md>)

Original publisher: [Read original article](<https://developers.googleblog.com/securing-your-agents-a-hands-on-guide-to-agent-anomaly-detection/>)

Author: Achuth Narayan Rajagopal; Derek Luan

Published: 2026-11-17T05:43:32.994674Z

Content type: tutorial

Language: en

Sources: [Google Developers Blog](<https://devfeed.tech/sources/google-developers-blog.md>)

Topics: [agent observability](<https://devfeed.tech/topics/agent-observability.md>), [threat detection](<https://devfeed.tech/topics/threat-detection.md>), [Cloud Monitoring](<https://devfeed.tech/topics/cloud-monitoring.md>), [ai observability](<https://devfeed.tech/topics/ai-observability.md>), [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [Detection engineering](<https://devfeed.tech/topics/detection-engineering.md>), [alert triage](<https://devfeed.tech/topics/alert-triage.md>), [google-adk](<https://devfeed.tech/topics/google-adk.md>), [AI for customer service automation](<https://devfeed.tech/topics/ai-for-customer-service-automation.md>)

Tags: [adk](<https://devfeed.tech/tags/adk.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [alerts](<https://devfeed.tech/tags/alerts.md>), [anomaly-detection](<https://devfeed.tech/tags/anomaly-detection.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-monitoring](<https://devfeed.tech/tags/cloud-monitoring.md>), [configure](<https://devfeed.tech/tags/configure.md>), [customer](<https://devfeed.tech/tags/customer.md>), [customer-support](<https://devfeed.tech/tags/customer-support.md>), [deploy](<https://devfeed.tech/tags/deploy.md>), [detection](<https://devfeed.tech/tags/detection.md>), [gemini](<https://devfeed.tech/tags/gemini.md>), [guide](<https://devfeed.tech/tags/guide.md>), [monitor](<https://devfeed.tech/tags/monitor.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [platform](<https://devfeed.tech/tags/platform.md>), [sample](<https://devfeed.tech/tags/sample.md>), [security](<https://devfeed.tech/tags/security.md>), [telemetry](<https://devfeed.tech/tags/telemetry.md>)

### AI overview

Agent Anomaly Detection on the Gemini Platform analyzes agent traces and logs to flag sessions that stray beyond an agent's intended job. The guide demonstrates it with a deliberately over-permissive customer support agent using canned in-memory data: after telemetry and message logging are enabled, a normal session and a simulated payroll-data request are processed, and the latter is flagged for tool misuse. Setup requires a Cloud Logging bucket and Observability dataset in the same US or EU multi-region; monitored sessions are analyzed after setup, which takes about 30 minutes, and new sessions are scanned within about five minutes. The example finding was High severity with 95% probability, but detector results and severities can vary because the analysis models are non-deterministic. Cloud Monitoring metrics support anomaly and coverage alerts; Security Command Center can receive anomaly findings when enabled for the organization. The excerpt cautions against copying the intentionally unsafe demo tool into production.

### Source excerpt

This guide walks developers through setting up Agent Anomaly Detection on the Gemini Enterprise Agent Platform to monitor AI agent telemetry and catch behavioral risks. It demonstrates how to deploy a sample customer support agent, trigger out-of-scope security findings, and configure proactive alerts using Cloud Monitoring and Security Command Center.

## Where the Evidence Lives: Measuring Privacy and Diagnosability of Exported AI-Agent Traces

DevFeed: [Where the Evidence Lives: Measuring Privacy and Diagnosability of Exported AI-Agent Traces](<https://devfeed.tech/articles/where-the-evidence-lives-measuring-privacy-and-diagnosability-of-exported-ai-agent-traces-139298.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20389>)

Author: Chang Xu, Hao Wang, Gedi Zhou, Junyuan Tan

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [ai observability](<https://devfeed.tech/topics/ai-observability.md>), [Detection engineering](<https://devfeed.tech/topics/detection-engineering.md>), [Data Management](<https://devfeed.tech/topics/data-management.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [agent](<https://devfeed.tech/tags/agent.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [all](<https://devfeed.tech/tags/all.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [attack](<https://devfeed.tech/tags/attack.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [but](<https://devfeed.tech/tags/but.md>), [can](<https://devfeed.tech/tags/can.md>), [cases](<https://devfeed.tech/tags/cases.md>), [comparison](<https://devfeed.tech/tags/comparison.md>), [comparisons](<https://devfeed.tech/tags/comparisons.md>), [counts](<https://devfeed.tech/tags/counts.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [customer](<https://devfeed.tech/tags/customer.md>), [customers](<https://devfeed.tech/tags/customers.md>), [diagnose](<https://devfeed.tech/tags/diagnose.md>), [measuring](<https://devfeed.tech/tags/measuring.md>), [privacy](<https://devfeed.tech/tags/privacy.md>), [traces](<https://devfeed.tech/tags/traces.md>)

### AI overview

The benchmark tests whether exported AI-agent traces preserve enough evidence to repair faulty outputs while limiting customer identification, using synthetic multi-tenant workflows. Conventional redaction can remove repair-relevant evidence yet leave encoded identifiers recoverable. Two alternatives preserve ownership comparisons: explicit comparison facts and execution-scoped consistent pseudonyms. Tested identifier matchers identified no requested customer from either export; the original diagnostic program repaired cases with both, while an independently written program produced correct outputs in 39 of 42 faulty cases with comparison facts and 42 of 42 with pseudonyms. However, row counts remained identifying: with prior knowledge of customer row counts, an attacker uniquely identified the requested customer in 97 of 240 executions under each of 15 policies. Removing selected counters from the comparison-fact export stopped this attack while preserving the original program's repairs.

### Source excerpt

arXiv:2611.20389v1 Announce Type: new Abstract: Execution traces help diagnose AI-agent failures, but sharing them can expose customer identities. For example, detecting that an agent handled the wrong customer's records requires knowing whether the records belong to the requested customer, but not necessarily knowing who that customer is. Our executable benchmark measures what a trace export reveals and whether a proposed repair restores the correct output in synthetic multi-tenant workflows. Conventional redaction can remove evidence needed for repair while leaving encoded identifiers recoverable. Two exports preserve ownership comparisons by reporting the match as a comparison fact or by using pseudonyms consistent within each execution. Both preserve the original diagnostic program's repairs. The tested identifier matchers identify no requested customer from either export. An independently written program produces correct outputs on 39 of 42 faulty cases with comparison facts and on all 42 with execution-scoped pseudonyms. Row counts still expose identities. With prior knowledge of customers' row counts, an attacker uniquely identifies the requested customer in 97 of 240 executions under each of the 15 export policies. Removing selected counters from the comparison-fact export stops this attack and preserves the original program's repairs. The benchmark lets engineers test what their diagnostic program can repair and what an attacker can still infer from the same export.

## Objectify, Don't Delete: Complaint-Derived Evaluation of Ambient Clinical Notes

DevFeed: [Objectify, Don't Delete: Complaint-Derived Evaluation of Ambient Clinical Notes](<https://devfeed.tech/articles/objectify-don-t-delete-complaint-derived-evaluation-of-ambient-clinical-notes-137454.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.19525>)

Author: Nikita Gupta, Narayanan Unny Edakunni, Vikram Okaly, Suranjana Datta, Navya Penubadi, Karl Eric Swanson, Siddharth Sharma

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [AI-generated responses](<https://devfeed.tech/topics/ai-generated-responses.md>), [DeepFact-Bench](<https://devfeed.tech/topics/deepfact-bench.md>), [ai observability](<https://devfeed.tech/topics/ai-observability.md>)

Tags: [aci](<https://devfeed.tech/tags/aci.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ambient](<https://devfeed.tech/tags/ambient.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [grounding](<https://devfeed.tech/tags/grounding.md>), [hallucinations](<https://devfeed.tech/tags/hallucinations.md>), [llm](<https://devfeed.tech/tags/llm.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [note](<https://devfeed.tech/tags/note.md>), [notes](<https://devfeed.tech/tags/notes.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [quality](<https://devfeed.tech/tags/quality.md>), [question](<https://devfeed.tech/tags/question.md>), [questions](<https://devfeed.tech/tags/questions.md>), [raise](<https://devfeed.tech/tags/raise.md>), [recall](<https://devfeed.tech/tags/recall.md>), [reference](<https://devfeed.tech/tags/reference.md>), [rouge](<https://devfeed.tech/tags/rouge.md>), [similarity](<https://devfeed.tech/tags/similarity.md>)

### AI overview

The paper argues that broad clinician complaints often combine several questions, making them hard for yes/no LLM judges to enforce. Its objectification step rewrites them as scoped decisions and evaluates defects per span. Across eleven planted defects in ACI-bench notes, objectified judges reached median recall of 0.95; reference metrics changed by at most 0.037. Asking the broad complaint as one question still found all planted defects but falsely flagged 70% of unperturbed notes. Asking one section-pair question per call raised Youden's J from 0.30 to 0.72, including with prompt effort matched. The authors also report clinician-reviewed labels identifying hallucinations in 32 of 40 gold-standard notes and omitted discussed facts in all 40. In 1,446 production notes, instance-rate burden showed a small association with clinician ratings (rho = -0.15). The paper cautions against collapsing span-level defect counts into a single note-level pass/fail score.

### Source excerpt

arXiv:2611.19525v1 Announce Type: new Abstract: Evaluating the quality of AI-generated clinical notes is hard: standard metrics like ROUGE measure lexical similarity to reference notes, while expert checklists encode what committees believe matters. Neither captures what clinicians actually complain about. Recent work has shown that evaluation criteria can be mined from user feedback and filtered to criteria that LLM judges can reliably enforce. We argue that this is where much of the value is lost. The criteria that fail enforceability are disproportionately the ones clinicians raise most often, because high-frequency complaints are stated as several questions glued together. No yes/no judgment can grade such composites as posed. We introduce an objectification step that rewrites such criteria into sets of scoped, determinate decisions. We demonstrate the approach through an evaluation pipeline that mines criteria from complaints, objectifies them, and deploys LLM judges against both planted defects and production notes. Across eleven defects planted into ACI-bench reference notes, objectified judges localize violations with median recall of 0.95, while reference metrics change by at most 0.037. Given the complaint as one broad question, the same judges still find every planted defect, but they also flag 70% of the unperturbed reference notes; one section-pair question per call raises Youden's J from 0.30 to 0.72 (by 0.19 to 0.26 with prompt effort matched). We release ACI-Defect, a perturbation benchmark; clinician-reviewed instance-level labels showing quality gaps in the ACI-bench gold standard (32 of 40 notes contain hallucinations; 40 of 40 omit discussed facts); and grounding-judge verdicts. On 1,446 production notes, mean instance-rate burden falls at every clinician-rating step (rho=-0.15), a small association. Defect counts have to be kept per span rather than collapsed into one pass/fail bit per note.

## Three Invisible Failure Modes in Speech-Based Age Estimation for Child Safety

DevFeed: [Three Invisible Failure Modes in Speech-Based Age Estimation for Child Safety](<https://devfeed.tech/articles/three-invisible-failure-modes-in-speech-based-age-estimation-for-child-safety-137370.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18971>)

Author: Christopher M. Stewart, Kelsey Kraus, J. Elliott Casal, Geoffrey L. Brown

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [LLM security](<https://devfeed.tech/topics/llm-security.md>), [reliability-engineering](<https://devfeed.tech/topics/reliability-engineering.md>), [ai observability](<https://devfeed.tech/topics/ai-observability.md>)

Tags: [age-assurance](<https://devfeed.tech/tags/age-assurance.md>), [ai](<https://devfeed.tech/tags/ai.md>), [child-safety](<https://devfeed.tech/tags/child-safety.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-cy](<https://devfeed.tech/tags/cs-cy.md>), [cs-sd](<https://devfeed.tech/tags/cs-sd.md>), [failure](<https://devfeed.tech/tags/failure.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [speech](<https://devfeed.tech/tags/speech.md>)

### AI overview

Across five studies using two speech corpora of children under 13, three audio-capable language models classified children as over 13 in 2.5%-42.5% of responses; every model that returned an age placed some 5- to 11-year-olds over 18. The study identifies pipeline failures that standard monitoring would miss: a client library silently double-encoded audio, prompting implausible age estimates and hallucinated explanations; a replaced endpoint refused nearly all age questions; and models sometimes gave inconsistent age classifications, including threshold answers that contradicted numeric estimates. None triggered warnings, status codes, or output-format changes. The authors argue that child-safety age-assurance claims need pipeline-level evaluation and reporting practices that expose these failures.

### Source excerpt

arXiv:2611.18971v1 Announce Type: new Abstract: Child safety in AI depends on the accurate identification of minor users. Young children tend to talk rather than type when interacting with AI, so in practice this identification requires predicting age from speech. We evaluated whether three audio-capable large language models can predict a child's age from their speech across five studies using two speech corpora of children under the age of 13. Models classified children as over 13 on 2.5% to 42.5% of audio responses, and every model that answered placed some 5- to 11-year-olds over 18. While measuring these rates, we uncovered three distinct failure modes. A client library silently double-encoded speech samples, leading one model to classify children aged 5 to 10 years old as being in their thirties and forties with hallucinated explanations. A second model's endpoint was replaced by a successor that refused nearly every age-related question and never produced an age. Every model sometimes placed the same speech sample under 13 and over 18 years of age, but a third model went further, giving threshold answers that contradicted its own numeric estimates. None of these failure modes produced a warning, a status code, or a change in output format, so they would all pass standard monitoring. These results demonstrate why child-safety claims about age assurance should be evaluated at the level of the pipeline rather than the model. We describe the reporting practices that would make such failures detectable.

## Language-model agents' beliefs can be manipulated via truthful evidence in open channel

DevFeed: [Language-model agents' beliefs can be manipulated via truthful evidence in open channel](<https://devfeed.tech/articles/language-model-agents-beliefs-can-be-manipulated-via-truthful-evidence-in-open-channel-137669.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21235>)

Author: Jinwei Hu, Qisong He, Xinmiao Huang, Youcheng Sun, Yi Dong, Xiaowei Huang

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [LLM security](<https://devfeed.tech/topics/llm-security.md>), [ai observability](<https://devfeed.tech/topics/ai-observability.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-ma](<https://devfeed.tech/tags/cs-ma.md>), [cs-si](<https://devfeed.tech/tags/cs-si.md>), [decision](<https://devfeed.tech/tags/decision.md>), [fragment](<https://devfeed.tech/tags/fragment.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [monitor](<https://devfeed.tech/tags/monitor.md>), [overfitting](<https://devfeed.tech/tags/overfitting.md>), [runtime](<https://devfeed.tech/tags/runtime.md>), [social-media](<https://devfeed.tech/tags/social-media.md>), [support](<https://devfeed.tech/tags/support.md>), [twitter](<https://devfeed.tech/tags/twitter.md>)

### AI overview

Language-model agents can accept and spread misleading rumours even when the fragments they receive are truthful: agents may infer relationships between fragments and treat those inferences as established facts. In tests on 1,090 rumour targets from six Twitter events, acceptance exceeded 65% in 12 of 14 models and remained above 50% after majority voting or review of the same evidence. The authors introduce Grounded Monitor, a proof-of-concept neuro-symbolic runtime monitor that checks whether conclusions have direct evidential support; it improved decision accuracy by 5.6 to 29.0 percentage points in original and anonymized settings. The results are reported in the supplied abstract.

### Source excerpt

arXiv:2611.21235v1 Announce Type: new Abstract: Language-model agents' beliefs can be manipulated through open channels such as social media, even when the information they receive is truthful. When agents fill gaps between fragments with self-induced relationships, they may accept those relationships as established facts and pass the resulting rumours to others. Drawing on cognitive theory of narrative understanding and film theory of montage, we introduce Generative Montage to operationalize this failure, termed narrative overfitting, by coordinating agents to arrange truthful fragments into misleading narratives. Across 1,090 rumour targets from six Twitter events, agents often accepted rumours overconfidently, with overall acceptance exceeding 65 percent in twelve of fourteen models. Acceptance remained above 50 percent after majority voting or review of the same evidence, showing how errors can reach downstream decisions. To explain this susceptibility, fragment interventions identify posts that influence belief within complete narratives. Synthetic controls then show that similar response patterns arise from valid evidence, separating belief influence from evidential support. This distinction motivates Grounded Monitor, a proof-of-concept neuro-symbolic runtime monitor that checks direct evidential support to guide agents' judgements, improving decision accuracy by 5.6 to 29.0 percentage points across original and anonymized settings. These findings expose a cognitive route to rumour amplification and support monitoring inferred relationships before agent-generated conclusions guide public decisions.

## But what is your honest answer? Aiding LLM-judges with honest alternatives using steering vectors

DevFeed: [But what is your honest answer? Aiding LLM-judges with honest alternatives using steering vectors](<https://devfeed.tech/articles/but-what-is-your-honest-answer-aiding-llm-judges-with-honest-alternatives-using-steering-vectors-138715.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2505.17760>)

Author: Leon Eshuijs, Archie Chaudhury, Alan McBeth, Ethan Nguyen

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [ai observability](<https://devfeed.tech/topics/ai-observability.md>)

Tags: [claude](<https://devfeed.tech/tags/claude.md>), [claude-haiku](<https://devfeed.tech/tags/claude-haiku.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [example](<https://devfeed.tech/tags/example.md>), [framework](<https://devfeed.tech/tags/framework.md>), [gpt](<https://devfeed.tech/tags/gpt.md>), [inference](<https://devfeed.tech/tags/inference.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-as-a-judge](<https://devfeed.tech/tags/llm-as-a-judge.md>), [manipulation](<https://devfeed.tech/tags/manipulation.md>), [pairs](<https://devfeed.tech/tags/pairs.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [response](<https://devfeed.tech/tags/response.md>), [reverse](<https://devfeed.tech/tags/reverse.md>), [safety](<https://devfeed.tech/tags/safety.md>), [steering](<https://devfeed.tech/tags/steering.md>), [uncertainty](<https://devfeed.tech/tags/uncertainty.md>), [vector](<https://devfeed.tech/tags/vector.md>), [white-box](<https://devfeed.tech/tags/white-box.md>)

### AI overview

JUSSA uses an honesty-promoting steering vector, optimized from one training example, to generate a contrastive alternative from the assessed model's internal representations; an LLM judge compares the response against that reference. On a human-validated manipulation dataset, AUROC rose from 0.893 to 0.946 for GPT-4.1 and from 0.859 to 0.929 for Claude Haiku, with gains concentrated on pairs judges otherwise misranked. The gains reversed when a judge had saturated the task and for the weakest judge tested, suggesting the method depends on a judge being able to use the reference without already reaching its task ceiling. Layer-wise analysis placed steering effectiveness in the middle layers. The abstract presents steering vectors as evaluation aids for white-box auditing, rather than as a way to improve outputs at inference.

### Source excerpt

arXiv:2505.17760v4 Announce Type: replace Abstract: LLM-as-a-judge is widely used as a scalable substitute for human evaluation, yet judges are least reliable on subtle dishonesty such as sycophancy and manipulation, where a judge scoring a response in isolation lacks a reference point. We introduce Judge Using Safety-Steered Alternatives (JUSSA), a framework that reduces this evaluator uncertainty using the assessed model's own internal representations. An honesty-promoting steering vector, optimized from a single training example, generates a contrastive alternative against which each response is judged. On a new manipulation dataset with human-validated response pairs, JUSSA improves AUROC for both GPT-4.1 (0.893$\rightarrow$0.946) and Claude Haiku (0.859$\rightarrow$0.929) judges. The gains concentrate almost entirely on the pairs a judge would otherwise misrank. They reverse once a judge has saturated a task and for the weakest judge we test, so contrastive evaluation helps most when a judge is capable enough to use the reference but has not yet saturated the task. A layer-wise analysis further locates steering effectiveness in the middle layers, where model representations of honest and dishonest prompt processing diverge. Our work demonstrates that steering vectors can serve as tools for evaluation rather than for improving model outputs at inference, opening a new direction for white-box auditing.

## The Biggest Risk to Your AI Is Not in the Chat Box, and Your LLM Firewall Cannot See It

DevFeed: [The Biggest Risk to Your AI Is Not in the Chat Box, and Your LLM Firewall Cannot See It](<https://devfeed.tech/articles/the-biggest-risk-to-your-ai-is-not-in-the-chat-box-and-your-llm-firewall-cannot-see-it-136091.md>)

Original publisher: [Read original article](<https://montecarlo.ai/blog-llm-firewall>)

Author: Ruslan Sultanov

Published: 2026-11-16T18:12:53Z

Content type: article

Language: en

Sources: [Monte Carlo](<https://devfeed.tech/sources/monte-carlo.md>)

Topics: [LLM security](<https://devfeed.tech/topics/llm-security.md>), [ai observability](<https://devfeed.tech/topics/ai-observability.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [human review](<https://devfeed.tech/topics/human-review.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agent-observability](<https://devfeed.tech/tags/agent-observability.md>), [agent-trust](<https://devfeed.tech/tags/agent-trust.md>), [agent-trust-glossary](<https://devfeed.tech/tags/agent-trust-glossary.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [api](<https://devfeed.tech/tags/api.md>), [api-keys](<https://devfeed.tech/tags/api-keys.md>), [background-jobs](<https://devfeed.tech/tags/background-jobs.md>), [bayes](<https://devfeed.tech/tags/bayes.md>), [chatbot](<https://devfeed.tech/tags/chatbot.md>), [cisa](<https://devfeed.tech/tags/cisa.md>), [firewall](<https://devfeed.tech/tags/firewall.md>), [glossary](<https://devfeed.tech/tags/glossary.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-firewall](<https://devfeed.tech/tags/llm-firewall.md>)

### AI overview

An LLM firewall can inspect messages to and from a model, but it may miss stale or poisoned data retrieved by an agent and cannot stop risky actions the agent is authorized to take. The excerpt recommends routing every model call through the firewall, scanning inputs and outputs, testing policies in log-only mode before enforcement, and accounting for latency from output scanning. It also recommends limiting each agent's permissions, logging model inputs, outputs, and tool or API calls, monitoring tool-use patterns and false positives, and using run traces plus data lineage to investigate incidents. Sudden behavior changes may indicate tampering; gradual declines may reflect data drift. Monte Carlo's described approach combines data and agent observability with lineage to track sources, behavior, and outputs; the excerpt does not establish that these measures guarantee prevention.

### Source excerpt

An LLM firewall has a simple job. It sits between the people using your AI agent and the large language model (LLM) behind it, checking every message going in and every reply coming out so that it can block anything that looks like an attack, a leaked password, or an off-limits topic. If someone types ...

## Introducing the Databricks AI Index: The state of enterprise AI tokenomics

DevFeed: [Introducing the Databricks AI Index: The state of enterprise AI tokenomics](<https://devfeed.tech/articles/introducing-the-databricks-ai-index-the-state-of-enterprise-ai-tokenomics-136418.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/introducing-databricks-ai-index-state-enterprise-ai-tokenomics>)

Author: The Databricks AI Product and Engineering Team

Published: 2026-11-16T17:00:00Z

Content type: release

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [databricks](<https://devfeed.tech/topics/databricks.md>), [AI-assisted coding](<https://devfeed.tech/topics/ai-assisted-coding.md>), [ai observability](<https://devfeed.tech/topics/ai-observability.md>), [Frontier AI](<https://devfeed.tech/topics/frontier-ai.md>)

Tags: [adoption](<https://devfeed.tech/tags/adoption.md>), [agent-harnesses](<https://devfeed.tech/tags/agent-harnesses.md>), [agent-tasks](<https://devfeed.tech/tags/agent-tasks.md>), [agent-workflows](<https://devfeed.tech/tags/agent-workflows.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-adoption](<https://devfeed.tech/tags/ai-adoption.md>), [ai-assistant](<https://devfeed.tech/tags/ai-assistant.md>), [ai-costs](<https://devfeed.tech/tags/ai-costs.md>), [ai-labs](<https://devfeed.tech/tags/ai-labs.md>), [ai-models](<https://devfeed.tech/tags/ai-models.md>), [ai-research](<https://devfeed.tech/tags/ai-research.md>), [coding](<https://devfeed.tech/tags/coding.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [databricks-ai](<https://devfeed.tech/tags/databricks-ai.md>), [developers](<https://devfeed.tech/tags/developers.md>), [enterprise-ai](<https://devfeed.tech/tags/enterprise-ai.md>), [index](<https://devfeed.tech/tags/index.md>), [models](<https://devfeed.tech/tags/models.md>), [platform](<https://devfeed.tech/tags/platform.md>), [tokenomics](<https://devfeed.tech/tags/tokenomics.md>), [unity-gateway](<https://devfeed.tech/tags/unity-gateway.md>)

### AI overview

The Databricks AI Index combines anonymized Unity Gateway usage and spending data from 10,000+ companies with weekly coding-agent benchmarks using Databricks workflows and hidden tests. In its September 21-November 7, 2026 dataset, enterprise token use shifted across providers and versions, open-weight models grew from 7.3% to 30.9% of traffic, and multi-provider use increased. Benchmark results show that model and harness combinations can differ sharply in task success and cost: GPT-6 Luna and Claude Opus 5.5 both scored 89.1%, while average task costs were $0.05 and $1.62. Low token prices do not guarantee low task costs, since extra turns and failed attempts add work. The Index is updated weekly; results reflect evaluated workloads and models and may vary by organization.

### Source excerpt

Developers have more AI models and coding agents to choose from than ever before....

## Our first phase of embedded assessments

DevFeed: [Our first phase of embedded assessments](<https://devfeed.tech/articles/our-first-phase-of-embedded-assessments-136430.md>)

Original publisher: [Read original article](<https://openai.com/index/our-first-phase-of-embedded-assessments>)

Published: 2026-11-16T08:00:00Z

Content type: news

Language: en

Sources: [OpenAI News](<https://devfeed.tech/sources/openai-news.md>)

Topics: [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [test](<https://devfeed.tech/topics/test.md>), [ai observability](<https://devfeed.tech/topics/ai-observability.md>), [post-training](<https://devfeed.tech/topics/post-training.md>)

Tags: [alignment](<https://devfeed.tech/tags/alignment.md>), [apollo](<https://devfeed.tech/tags/apollo.md>), [assessment](<https://devfeed.tech/tags/assessment.md>), [effectiveness](<https://devfeed.tech/tags/effectiveness.md>), [embedded](<https://devfeed.tech/tags/embedded.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [evaluations](<https://devfeed.tech/tags/evaluations.md>), [experts](<https://devfeed.tech/tags/experts.md>), [first](<https://devfeed.tech/tags/first.md>), [metr](<https://devfeed.tech/tags/metr.md>), [model-alignment](<https://devfeed.tech/tags/model-alignment.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [openai](<https://devfeed.tech/tags/openai.md>), [safety](<https://devfeed.tech/tags/safety.md>), [transluce](<https://devfeed.tech/tags/transluce.md>)

### AI overview

OpenAI has begun embedded safety assessments with Apollo Research, METR, Transluce, and Goodfire, giving independent experts access to selected internal models, checkpoints, data, monitoring systems, and staff. Apollo will examine possible covert subversion and whether safeguards detect it; Goodfire will use interpretability methods for training monitoring, debugging, and alignment; METR will assess misalignment in a large reinforcement learning run, with a broader monitor assessment still pending agreement; Transluce will review post-training alignment evaluations and may conduct a broader follow-on assessment in early 2027. Each engagement has an agreed scope and safety arrangements. Assessors can refine methods and request additional access, with alternatives considered for sensitive material. The Safety and Security Committee receives unredacted reports and can speak privately with assessors. Assessors retain publication independence, while OpenAI may redact confidential information or request delays for safety or security remediation. The article says these early protocols may evolve and identifies access, space, and security challenges.

### Source excerpt

OpenAI begins embedded assessments with Apollo Research, METR, and Transluce, and Goodfire to independently evaluate model alignment, safeguards, and monitoring systems.

## Mapping agent controls to the OWASP Top 10 for LLM apps

DevFeed: [Mapping agent controls to the OWASP Top 10 for LLM apps](<https://devfeed.tech/articles/mapping-agent-controls-to-the-owasp-top-10-for-llm-apps-134659.md>)

Original publisher: [Read original article](<https://nirmata.com/2026/11/16/mapping-agent-controls-to-owasp-llm-top-10/>)

Author: Ritesh Patel

Published: 2026-11-16T05:03:28.562711Z

Content type: article

Language: en

Sources: [Nirmata](<https://devfeed.tech/sources/nirmata.md>)

Topics: [LLM security](<https://devfeed.tech/topics/llm-security.md>), [human in the loop agent](<https://devfeed.tech/topics/human-in-the-loop-agent.md>), [ai observability](<https://devfeed.tech/topics/ai-observability.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [ai-governance](<https://devfeed.tech/tags/ai-governance.md>), [apps](<https://devfeed.tech/tags/apps.md>), [controls](<https://devfeed.tech/tags/controls.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [limit](<https://devfeed.tech/tags/limit.md>), [llm](<https://devfeed.tech/tags/llm.md>), [logs](<https://devfeed.tech/tags/logs.md>), [map](<https://devfeed.tech/tags/map.md>), [mapping](<https://devfeed.tech/tags/mapping.md>), [model](<https://devfeed.tech/tags/model.md>), [owasp](<https://devfeed.tech/tags/owasp.md>), [owasp-llm-top-10](<https://devfeed.tech/tags/owasp-llm-top-10.md>), [owasp-top-10](<https://devfeed.tech/tags/owasp-top-10.md>), [policy](<https://devfeed.tech/tags/policy.md>), [risks](<https://devfeed.tech/tags/risks.md>), [scope](<https://devfeed.tech/tags/scope.md>), [security](<https://devfeed.tech/tags/security.md>), [tool](<https://devfeed.tech/tags/tool.md>)

### AI overview

The post maps the OWASP LLM Top 10 2026 to controls for limiting agent risk, while excluding OWASP's separate agentic risk list. It highlights tool scoping and least privilege, authorization checks outside the model, approval for high impact or irreversible actions, hard spend and execution limits, and contextual logs. Its review checklist calls for checking tool permissions, allow decisions, approval needs, limits that halt runs, package and description changes, and adaptive attack testing. The post stresses that prompt injection has no reliable prevention mechanism, so controls should constrain what a compromised model can do; pinning tools alone does not prevent malicious payloads or poisoned descriptions. Logs can themselves expose sensitive data and should be restricted and scrubbed. It also describes four AIControls product capabilities as examples, while presenting the mapping as vendor neutral and a summary of OWASP mitigations.

### Source excerpt

Map the OWASP LLM Top 10 2026 risks to the agent controls that limit them: tool scope, policy checks outside the model, approval, spend caps and logs.

## Interpreting Coding Agents Through Their Behavioral Trajectories

DevFeed: [Interpreting Coding Agents Through Their Behavioral Trajectories](<https://devfeed.tech/articles/interpreting-coding-agents-through-their-behavioral-trajectories-135065.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.17440>)

Author: Ibragim Badertdinov, Alexander Golubev

Published: 2026-11-16T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [agentic-coding](<https://devfeed.tech/topics/agentic-coding.md>), [AI-assisted coding](<https://devfeed.tech/topics/ai-assisted-coding.md>), [ai observability](<https://devfeed.tech/topics/ai-observability.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [aggregate](<https://devfeed.tech/tags/aggregate.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [annotation](<https://devfeed.tech/tags/annotation.md>), [audit](<https://devfeed.tech/tags/audit.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [coding](<https://devfeed.tech/tags/coding.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [execution](<https://devfeed.tech/tags/execution.md>), [explore](<https://devfeed.tech/tags/explore.md>), [file-editing](<https://devfeed.tech/tags/file-editing.md>), [frontier-models](<https://devfeed.tech/tags/frontier-models.md>), [languages](<https://devfeed.tech/tags/languages.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [test](<https://devfeed.tech/tags/test.md>), [testing](<https://devfeed.tech/tags/testing.md>), [token](<https://devfeed.tech/tags/token.md>), [tool](<https://devfeed.tech/tags/tool.md>)

### AI overview

The study analyzes 143,209 coding-agent execution trajectories from 125 configurations spanning 117 models, using a rule-based parser to classify actions such as searching repositories, editing files, and running tests. Its audit compared parser predictions with a six-model consensus on 500 events. Agents with similar task success rates showed substantially different exploration, editing, and verification behaviors. Aggregate behavioral profiles could identify agent stacks across separate repositories and programming languages without using outcomes, token counts, or tool names, though task composition was not fully controlled in that comparison. The authors also report that scaffold choice and reasoning effort systematically affect search, editing, and testing behavior, and release the parser, analysis pipeline, and an annotated trajectory subset.

### Source excerpt

arXiv:2611.17440v1 Announce Type: cross Abstract: Standard coding-agent benchmarks primarily evaluate binary task success, treating execution as a black box and obscuring the behaviors that produce those outcomes. We analyze a large corpus of 143,209 execution trajectories from 125 agent configurations spanning 117 unique models. To enable comparisons across heterogeneous agent interfaces, we develop a rule-based parser that maps raw execution traces to high-level actions such as repository search, file editing, and test execution, and audit its predictions against a six-model consensus reference on 500 events. We find that agents with similar solve rates can nevertheless differ substantially in how they explore, modify and verify code. Moreover, without using task outcomes, token counts, or tool names as features, aggregate behavioral profiles retrieve the corresponding agent stack across disjoint repository sets and programming languages, although task composition is not fully controlled in the cross-repository setting. Through analyses of frontier models and agent scaffolds, we further show that scaffold choice and reasoning effort systematically shift agents' search, editing, and testing behavior. We release the deterministic parser together with the full annotation and analysis pipeline alongside an annotated trajectory subset.

## Do LLMs Experience an Internal Polylogue? Investigating Reasoning through the Lens of Personas

DevFeed: [Do LLMs Experience an Internal Polylogue? Investigating Reasoning through the Lens of Personas](<https://devfeed.tech/articles/do-llms-experience-an-internal-polylogue-investigating-reasoning-through-the-lens-of-personas-134980.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.09159>)

Author: Nils A. Herrmann, Leander Girrbach, Kirill Bykov, Zeynep Akata

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [ai observability](<https://devfeed.tech/topics/ai-observability.md>)

Tags: [activation](<https://devfeed.tech/tags/activation.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [generation](<https://devfeed.tech/tags/generation.md>), [hidden](<https://devfeed.tech/tags/hidden.md>), [intervene](<https://devfeed.tech/tags/intervene.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [linear](<https://devfeed.tech/tags/linear.md>), [llms](<https://devfeed.tech/tags/llms.md>), [models](<https://devfeed.tech/tags/models.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [open-weight-models](<https://devfeed.tech/tags/open-weight-models.md>), [persona](<https://devfeed.tech/tags/persona.md>), [persona-vectors](<https://devfeed.tech/tags/persona-vectors.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [robust](<https://devfeed.tech/tags/robust.md>), [stage](<https://devfeed.tech/tags/stage.md>), [steering](<https://devfeed.tech/tags/steering.md>), [time-series](<https://devfeed.tech/tags/time-series.md>), [tool](<https://devfeed.tech/tags/tool.md>)

### AI overview

The authors define "polylogue" as the changing alignment between persona directions in a model's activations and its hidden activations during generation. Across four open-weight models, these signals predict MMLU-Pro correctness about as well as low-dimensional activation baselines, while retaining an interpretable link to persona directions. The signals also suggest which latent directions to steer at different response stages. A paragraph-conditioned intervention improves accuracy on three models but reduces it on the fourth, so the reported stage-aware steering approach is not yet robust.

### Source excerpt

arXiv:2605.09159v3 Announce Type: replace Abstract: Recent work shows that large language models (LLMs) encode behavioural traits ("personas") as linear directions in activation space, often called "persona vectors". Prior work has used such directions as static handles for behavioural steering. We instead treat them as dynamic signals: probes we can monitor and intervene on as reasoning unfolds. We use the term polylogue to denote the time series of alignments between persona vectors and hidden activations over the course of generation. Experiments across four open-weight models show that polylogue features predict correctness on MMLU-Pro on par with low-dimensional activation baselines, while remaining interpretable through their associated persona directions. They also suggest concrete steering targets, namely which latent directions to modulate at different stages of a response. We instantiate this as a simple paragraph-conditioned intervention that improves accuracy on three of four models but degrades the fourth, suggesting that stage-aware latent steering is possible but not yet robust. Together, this positions the polylogue as an interpretable tool for reasoning-time monitoring and intervention.

## Uber's Secure AI Framework Governs Autonomous AI Agents Across Their Lifecycle

DevFeed: [Uber's Secure AI Framework Governs Autonomous AI Agents Across Their Lifecycle](<https://devfeed.tech/articles/uber-s-secure-ai-framework-governs-autonomous-ai-agents-across-their-lifecycle-134582.md>)

Original publisher: [Read original article](<https://www.infoq.com/news/2026/11/uber-secure-ai-framework/>)

Author: Sergio De Simone

Published: 2026-11-15T20:00:00Z

Content type: article

Language: en

Sources: [InfoQ](<https://devfeed.tech/sources/infoq.md>)

Topics: [LLM security](<https://devfeed.tech/topics/llm-security.md>), [ai observability](<https://devfeed.tech/topics/ai-observability.md>), [API Governance](<https://devfeed.tech/topics/api-governance.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [ai-framework](<https://devfeed.tech/tags/ai-framework.md>), [ai-guard](<https://devfeed.tech/tags/ai-guard.md>), [ai-ml-data-engineering](<https://devfeed.tech/tags/ai-ml-data-engineering.md>), [ai-security](<https://devfeed.tech/tags/ai-security.md>), [architecture-design](<https://devfeed.tech/tags/architecture-design.md>), [development](<https://devfeed.tech/tags/development.md>), [lifecycle](<https://devfeed.tech/tags/lifecycle.md>), [ml-data-engineering](<https://devfeed.tech/tags/ml-data-engineering.md>), [news](<https://devfeed.tech/tags/news.md>), [secure-ai](<https://devfeed.tech/tags/secure-ai.md>), [securing-ai](<https://devfeed.tech/tags/securing-ai.md>), [securing-ai-agents](<https://devfeed.tech/tags/securing-ai-agents.md>), [security](<https://devfeed.tech/tags/security.md>), [uber](<https://devfeed.tech/tags/uber.md>), [uber-secure-ai-framework](<https://devfeed.tech/tags/uber-secure-ai-framework.md>)

### AI overview

Uber's framework adds behavioral authorization to identity and API controls, checking whether an AI agent's proposed action fits the user's request, the agent's purpose and permissions, and the interaction context. It spans three stages: before deployment, agents are registered with identities, intents, constraints, dependencies, and risk classifications; Argus analyzes code and configuration for weaknesses, including for MCP servers and tools. At runtime, AI Guards assess interactions and actions against registry policies, and Agent Guard hooks apply their verdicts at checkpoints. After deployment, execution traces link actions, tool calls, and security decisions; recorded interactions can be replayed against updated policies. Uber reports tradeoffs in policy strictness, latency, developer setup, and agent-specific controls. Commentary in the excerpt notes that tailored intents and constraints can be costly across workflows, while raising the question of enforcement for vendor agents without hooks. The framework is still evolving.

### Source excerpt

In a recent article, Uber Engineering highlights the limitations of traditional identity and API controls in securing AI agents that dynamically reason over external content and take real-world actions. Uber developed a framework to determine whether an agent's action is appropriate given the user's request, the agent's intended purpose, its permissions, and the context of the interaction. By Sergio De Simone

[Next page](<https://devfeed.tech/topics/ai-observability.md?cursor=WyIyMDI2LTExLTE1VDIwOjAwOjAwKzAwOjAwIiwgIjc2N2RiMWIzLTk5N2YtNDMzMi05ZGNhLWYwZGNkOGE4MmNjYiJd>)