# LLM observability

LLM observability is the practice of collecting and analyzing telemetry from large-language-model applications to monitor behavior, performance, output quality, cost, safety, and reliability.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Datadog LLM Observability: Features, Pricing, and Alternatives

DevFeed: [Datadog LLM Observability: Features, Pricing, and Alternatives](<https://devfeed.tech/articles/datadog-llm-observability-features-pricing-and-alternatives-145962.md>)

Original publisher: [Read original article](<https://newrelic.com/blog/ai/datadog-llm-observability>)

Author: John Blust

Published: 2026-11-20T11:51:52.599709Z

Content type: article

Language: en

Sources: [New Relic](<https://devfeed.tech/sources/new-relic.md>)

Topics: [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [datadog](<https://devfeed.tech/topics/datadog.md>), [agent observability](<https://devfeed.tech/topics/agent-observability.md>), [ai observability](<https://devfeed.tech/topics/ai-observability.md>), [observability](<https://devfeed.tech/topics/observability.md>), [tracing](<https://devfeed.tech/topics/tracing.md>), [OpenTelemetry](<https://devfeed.tech/topics/opentelemetry.md>), [LLM security](<https://devfeed.tech/topics/llm-security.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [phoenix](<https://devfeed.tech/topics/phoenix.md>), [LangChain](<https://devfeed.tech/topics/langchain.md>), [dynatrace](<https://devfeed.tech/topics/dynatrace.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-monitoring](<https://devfeed.tech/tags/ai-monitoring.md>), [alternatives](<https://devfeed.tech/tags/alternatives.md>), [compare](<https://devfeed.tech/tags/compare.md>), [datadog](<https://devfeed.tech/tags/datadog.md>), [devops](<https://devfeed.tech/tags/devops.md>), [evaluations](<https://devfeed.tech/tags/evaluations.md>), [features](<https://devfeed.tech/tags/features.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-observability](<https://devfeed.tech/tags/llm-observability.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [new-relic](<https://devfeed.tech/tags/new-relic.md>), [observability](<https://devfeed.tech/tags/observability.md>), [pricing](<https://devfeed.tech/tags/pricing.md>), [tracing](<https://devfeed.tech/tags/tracing.md>)

### AI overview

Datadog LLM Observability, now called Agent Observability, traces model calls and agent workflows, evaluates quality and safety, supports prompt experiments, and tracks latency, tokens, errors, and costs. It is a standalone product billed by model-call spans; tool, retrieval, and workflow spans do not count. Both listed plans include core tracing and evaluation features, and annual Pro pricing starts at $160 per month, with retention add-ons affecting per-span rates. Alternatives differ in approach: New Relic connects AI telemetry to existing APM, including optional eBPF monitoring; Langfuse offers a self-hostable core; LangSmith emphasizes agent tracing and evaluation workflows; and Arize Phoenix supports local deployment with OpenTelemetry-based instrumentation. The text notes that Datadog can correlate with its other products through shared trace IDs and supports Python, Node.js, Java, OpenTelemetry, and HTTP API integration.

### Source excerpt

An objective look at Datadog LLM Observability: tracing, evaluations, how pricing works, and how alternatives like New Relic AI Monitoring compare.

## What the model knows but does not say: monitoring medical LLMs through verbalized thoughts

DevFeed: [What the model knows but does not say: monitoring medical LLMs through verbalized thoughts](<https://devfeed.tech/articles/what-the-model-knows-but-does-not-say-monitoring-medical-llms-through-verbalized-thoughts-145455.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24994>)

Author: Frederik Hauke, Jan Clusmann, Hanna Kreutzer, Christiane Kuhl, Tianyu Han, Felix Barajas Ordonez, Sven Nebelung, Jakob Nikolas Kather, Daniel Truhn

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [LLM security](<https://devfeed.tech/topics/llm-security.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [autoencoder](<https://devfeed.tech/tags/autoencoder.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [gemma-3](<https://devfeed.tech/tags/gemma-3.md>), [llama-3-3](<https://devfeed.tech/tags/llama-3-3.md>), [llms](<https://devfeed.tech/tags/llms.md>), [medical](<https://devfeed.tech/tags/medical.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [thoughts](<https://devfeed.tech/tags/thoughts.md>)

### AI overview

The study reports that verbalized internal activations can reveal clinical prompt injections that ordinary answers conceal. A natural-language autoencoder converts Gemma-3-27B activations into text; an input-blind judge assesses that "thoughts" text and the model's answer separately. Across two attack objectives (omitting pathology or covertly recommending a drug), two input modalities, and two delivery mechanisms, the thoughts channel detected injections significantly more often in all eight conditions (p<0.05), while both channels had near-zero false positives on clean inputs. The omission finding also carried over to Llama-3.3-70B. The channel flagged unsafe requests while the model was still reading them. The approach used an off-the-shelf verbalizer and general-purpose judge, without retraining; the excerpt does not establish performance beyond these reported settings.

### Source excerpt

arXiv:2611.24994v1 Announce Type: cross Abstract: Efforts to keep clinical language models safe have concentrated on what enters and leaves the model, but a manipulated model can return an answer that looks entirely ordinary. Here we show that the model's own internal state, rendered in plain language, exposes manipulations the output conceals. Using a natural-language autoencoder, we translate the activations of a multimodal LLM (Gemma-3-27B) into text and let an input-blind judge read this verbalized internal state (the thoughts channel) and the model's answer separately. Across two clinical attack objectives (omitting a pathology, covertly recommending a drug), two input modalities, and two delivery mechanisms, the thoughts channel caught the injection significantly more often than the answer in all eight injected conditions (p<0.05), while both channels held near-zero false-positive rates on clean inputs; the omission result carried over to Llama-3.3-70B. The thoughts channel also registered unsafe clinical requests as harmful while the model was still reading them, before it decided whether to refuse. None of this requires retraining: an off-the-shelf verbalizer and a general-purpose judge are enough.

## Persistent Sparse Autoencoders: Learning Feature-Specific Timescales in Language Model Representations

DevFeed: [Persistent Sparse Autoencoders: Learning Feature-Specific Timescales in Language Model Representations](<https://devfeed.tech/articles/persistent-sparse-autoencoders-learning-feature-specific-timescales-in-language-model-representations-144914.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2607.17117>)

Author: Haoyan Luo, Mateo Espinosa Zarlenga, Mateja Jamnik

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Representation learning](<https://devfeed.tech/topics/representation-learning.md>), [Language models](<https://devfeed.tech/topics/language-models.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [autoencoders](<https://devfeed.tech/tags/autoencoders.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [error](<https://devfeed.tech/tags/error.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [extension](<https://devfeed.tech/tags/extension.md>), [fast](<https://devfeed.tech/tags/fast.md>), [feature](<https://devfeed.tech/tags/feature.md>), [features](<https://devfeed.tech/tags/features.md>), [hidden](<https://devfeed.tech/tags/hidden.md>), [hidden-states](<https://devfeed.tech/tags/hidden-states.md>), [information](<https://devfeed.tech/tags/information.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [learning](<https://devfeed.tech/tags/learning.md>), [monitor](<https://devfeed.tech/tags/monitor.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [persistence](<https://devfeed.tech/tags/persistence.md>), [prompt-injection](<https://devfeed.tech/tags/prompt-injection.md>), [quality](<https://devfeed.tech/tags/quality.md>), [reduce](<https://devfeed.tech/tags/reduce.md>)

### AI overview

Persistent Sparse Autoencoders (Persistent SAEs) learn a persistence coefficient for each sparse feature, giving features different timescales based on reconstruction. The authors report that standard SAE features can already persist: after activation, hidden states remain aligned with a feature direction, and past activations help reconstruct later states. Persistent SAEs retain competitive reconstruction quality while producing fast features that remain locally interpretable and slow features that accumulate higher-level context. In a prompt-injection monitoring case study, slow features retain injection signals over long contexts and causally affect monitor judgements. The abstract presents these results as opportunities for model interpretation and monitoring.

### Source excerpt

arXiv:2607.17117v3 Announce Type: replace Abstract: Sparse autoencoders (SAEs) decompose language model activations into sparse features, yet these models traditionally encode each token independently, failing to expose information that persists across a sequence. We first show that temporal persistence can naturally emerge in standard SAE features: after a feature activates, the hidden state remains aligned with its direction, and past activations reduce the reconstruction error of later hidden states. How long this lasts varies widely across features. We therefore introduce Persistent Sparse Autoencoders (Persistent SAEs), an extension of standard SAEs that learns a persistence coefficient for each feature, allowing feature-specific timescales to emerge from reconstruction. Our experiments show that Persistent SAEs retain competitive reconstruction quality while learning a spectrum of timescales: short-timescale (fast) features remain locally interpretable, whereas long-timescale (slow) features accumulate higher-level contextual information. Moreover, we show in a prompt-injection monitoring case study that slow features retain injection signals over long contexts and causally affect monitor judgements. These results suggest that Persistent SAEs offer new opportunities for interpreting and monitoring language models via persistent features.

## Why agent governance starts with observability

DevFeed: [Why agent governance starts with observability](<https://devfeed.tech/articles/why-agent-governance-starts-with-observability-146227.md>)

Original publisher: [Read original article](<https://www.langchain.com/blog/agent-governance-observability>)

Author: LangChain Accounts

Published: 2026-11-20T00:57:43Z

Content type: article

Language: en

Sources: [LangChain Blog](<https://devfeed.tech/sources/langchain-blog.md>)

Topics: [agent observability](<https://devfeed.tech/topics/agent-observability.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [ai-governance](<https://devfeed.tech/topics/ai-governance.md>), [AI Agent](<https://devfeed.tech/topics/ai-agent.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agent-observability](<https://devfeed.tech/tags/agent-observability.md>), [agent-security](<https://devfeed.tech/tags/agent-security.md>), [ai](<https://devfeed.tech/tags/ai.md>), [governance](<https://devfeed.tech/tags/governance.md>), [observability](<https://devfeed.tech/tags/observability.md>)

### AI overview

Agent governance depends on observability: centralized traces and run trajectories let teams see which agents are operating, inspect their actions, and check whether guardrails hold. The excerpt distinguishes preventative controls--runtime limits and policies, plus pre-deployment evaluations and red-teaming--from detective measures such as online evaluations, dashboards, alerts, drift checks, and human review. Traces support both: they help explain failures before release and audit production behavior, including alternate paths that may bypass intended restrictions. Production findings can feed back into tighter guardrails or regression tests. The article also names LangSmith Insights reports for clustering trace patterns and LangSmith Engine for analyzing recurring failures and testing proposed fixes. These are presented as vendor capabilities; the excerpt does not establish comparative performance against other tools.

### Source excerpt

A strong AI agent governance strategy

## How the BMW Group governs hundreds of agents across thousands of developers

DevFeed: [How the BMW Group governs hundreds of agents across thousands of developers](<https://devfeed.tech/articles/how-the-bmw-group-governs-hundreds-of-agents-across-thousands-of-developers-144472.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/machine-learning/how-the-bmw-group-governs-hundreds-of-agents-across-thousands-of-developers/>)

Author: Dr. Jens Kohl

Published: 2026-11-19T22:20:23Z

Content type: article

Language: en

Sources: [Artificial Intelligence](<https://devfeed.tech/sources/artificial-intelligence.md>)

Topics: [ai-governance](<https://devfeed.tech/topics/ai-governance.md>), [Multi Agent Systems](<https://devfeed.tech/topics/multi-agent-systems.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [Cloud Architecture](<https://devfeed.tech/topics/cloud-architecture.md>), [LLM security](<https://devfeed.tech/topics/llm-security.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [agent to agent communication](<https://devfeed.tech/topics/agent-to-agent-communication.md>), [OpenTelemetry](<https://devfeed.tech/topics/opentelemetry.md>), [internal developer platform automation](<https://devfeed.tech/topics/internal-developer-platform-automation.md>)

Tags: [2023](<https://devfeed.tech/tags/2023.md>), [a2a](<https://devfeed.tech/tags/a2a.md>), [a2a-protocol](<https://devfeed.tech/tags/a2a-protocol.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agent-harness](<https://devfeed.tech/tags/agent-harness.md>), [agent-infrastructure](<https://devfeed.tech/tags/agent-infrastructure.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [automotive](<https://devfeed.tech/tags/automotive.md>), [best-practices](<https://devfeed.tech/tags/best-practices.md>), [bmw](<https://devfeed.tech/tags/bmw.md>), [compliance](<https://devfeed.tech/tags/compliance.md>), [developers](<https://devfeed.tech/tags/developers.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [enterprise-scale](<https://devfeed.tech/tags/enterprise-scale.md>), [governance](<https://devfeed.tech/tags/governance.md>), [intermediate-200](<https://devfeed.tech/tags/intermediate-200.md>), [patterns](<https://devfeed.tech/tags/patterns.md>), [practices](<https://devfeed.tech/tags/practices.md>), [quality](<https://devfeed.tech/tags/quality.md>), [scale](<https://devfeed.tech/tags/scale.md>), [teams](<https://devfeed.tech/tags/teams.md>)

### AI overview

BMW Group's approach to governing agents across its R&D organization centers on reusable architecture and lifecycle controls that let teams experiment while enforcing safety, security, quality, and compliance. Governance rules live outside individual agents--in integration layers, a central registry, and deterministic CI/CD and policy checks--so they can survive framework or model changes. Teams set autonomy per workflow step, from explicit human approval to bounded autonomy, and increase it based on evaluation evidence. Agents are built from composable MCP tools, skills, and flows, with specialized agents and least-privilege access used to contain complexity. CI pipelines run tests, security and governance checks, and ground-truth evaluations before deployment; runtime monitoring flags data or model drift for human review. IaC templates, shared registries, and centrally curated checks support distributed team contributions without mandating a single framework. The article describes these practices and reported benefits but provides no quantitative outcome measures.

### Source excerpt

When hundreds of agents serve thousands of developers, governance cannot be an afterthought. The BMW Group shares the practices and patterns its R&D teams use to build, govern, and operate AI agents at enterprise scale without sacrificing speed, quality, or compliance.

## What Is Tool Calling? How LLMs and AI Agents Take Action

DevFeed: [What Is Tool Calling? How LLMs and AI Agents Take Action](<https://devfeed.tech/articles/what-is-tool-calling-how-llms-and-ai-agents-take-action-145886.md>)

Original publisher: [Read original article](<https://montecarlo.ai/blog-what-is-tool-calling>)

Author: Virna Sekuj

Published: 2026-11-19T16:58:24Z

Content type: article

Language: en

Sources: [Monte Carlo](<https://devfeed.tech/sources/monte-carlo.md>)

Topics: [function calling](<https://devfeed.tech/topics/function-calling.md>), [Multi-step AI processes](<https://devfeed.tech/topics/multi-step-ai-processes.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [API](<https://devfeed.tech/topics/api.md>), [Tool](<https://devfeed.tech/topics/tool.md>)

Tags: [2023](<https://devfeed.tech/tags/2023.md>), [2026](<https://devfeed.tech/tags/2026.md>), [actions](<https://devfeed.tech/tags/actions.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agent-behavior](<https://devfeed.tech/tags/agent-behavior.md>), [agent-frameworks](<https://devfeed.tech/tags/agent-frameworks.md>), [agent-trust](<https://devfeed.tech/tags/agent-trust.md>), [agent-trust-glossary](<https://devfeed.tech/tags/agent-trust-glossary.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [api](<https://devfeed.tech/tags/api.md>), [database](<https://devfeed.tech/tags/database.md>), [function](<https://devfeed.tech/tags/function.md>), [function-calling](<https://devfeed.tech/tags/function-calling.md>), [glossary](<https://devfeed.tech/tags/glossary.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>), [llm-tool-calling](<https://devfeed.tech/tags/llm-tool-calling.md>), [llms](<https://devfeed.tech/tags/llms.md>), [querying](<https://devfeed.tech/tags/querying.md>), [searching](<https://devfeed.tech/tags/searching.md>), [structured](<https://devfeed.tech/tags/structured.md>), [tool-calling](<https://devfeed.tech/tags/tool-calling.md>), [tool-calling-examples](<https://devfeed.tech/tags/tool-calling-examples.md>), [tool-calling-vs-mcp](<https://devfeed.tech/tags/tool-calling-vs-mcp.md>), [tool-use-ai-agents](<https://devfeed.tech/tags/tool-use-ai-agents.md>), [uses](<https://devfeed.tech/tags/uses.md>)

### AI overview

Tool calling lets an LLM request an action by returning a structured tool name and arguments; the application validates and executes the request, sends back the result, and the model uses it to answer or make another call. Function calling and tool use are alternate names for the same mechanism. MCP standardizes how compatible applications discover and connect to tools; it builds on tool calling rather than replacing it. Calls can fail through wrong tool choice, bad arguments, skipped checks, retries, mishandled errors, changed responses, or unintended access, often without obvious errors. The text recommends tracing each call and its result, validating arguments and permissions before execution, watching for unexpected patterns and loops, tracking cost and latency, monitoring the underlying data, and limiting or requiring approval for high-impact actions.

### Source excerpt

Tool calling, also called function calling, is how a large language model asks an application to run a function, such as querying a database, calling an API, or searching documents, and then uses the result in its answer. The model doesn't run the tool itself. It produces a structured request naming the tool and its arguments, the application executes it, and the result goes back to the model.

## What Is an LLM? Large Language Models Explained

DevFeed: [What Is an LLM? Large Language Models Explained](<https://devfeed.tech/articles/what-is-an-llm-large-language-models-explained-145885.md>)

Original publisher: [Read original article](<https://montecarlo.ai/blog-what-is-an-llm-large-language-models-explained>)

Author: Virna Sekuj

Published: 2026-11-19T16:43:33Z

Content type: article

Language: en

Sources: [Monte Carlo](<https://devfeed.tech/sources/monte-carlo.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>), [context window](<https://devfeed.tech/topics/context-window.md>), [knowledge cutoff](<https://devfeed.tech/topics/knowledge-cutoff.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [agent-trust](<https://devfeed.tech/tags/agent-trust.md>), [agent-trust-glossary](<https://devfeed.tech/tags/agent-trust-glossary.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [attention](<https://devfeed.tech/tags/attention.md>), [chatgpt](<https://devfeed.tech/tags/chatgpt.md>), [claude](<https://devfeed.tech/tags/claude.md>), [code-generation](<https://devfeed.tech/tags/code-generation.md>), [coding-assistants](<https://devfeed.tech/tags/coding-assistants.md>), [context](<https://devfeed.tech/tags/context.md>), [context-window](<https://devfeed.tech/tags/context-window.md>), [glossary](<https://devfeed.tech/tags/glossary.md>), [how-do-llms-work](<https://devfeed.tech/tags/how-do-llms-work.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-examples](<https://devfeed.tech/tags/llm-examples.md>), [llm-limitations](<https://devfeed.tech/tags/llm-limitations.md>), [llm-meaning](<https://devfeed.tech/tags/llm-meaning.md>), [llm-vs-ai-agent](<https://devfeed.tech/tags/llm-vs-ai-agent.md>), [neural-network](<https://devfeed.tech/tags/neural-network.md>)

### AI overview

LLMs generate text by predicting one token at a time, and their answers depend heavily on the context supplied at runtime. In production, fluent output can still be wrong because of hallucinations, stale or incomplete context, incorrect tool use, or provider model changes; failures may not show up in uptime or error rates. The excerpt recommends recording traces of prompts, retrieved data, tool calls, outputs, latency, token counts, and model versions; evaluating accuracy, relevance, and groundedness; checking context freshness and completeness; and tracking cost and performance. Comparing traces and data lineage can help distinguish a context problem from a model change. The excerpt does not cover full training stages in detail.

### Source excerpt

A large language model (LLM) is a neural network trained on large amounts of text to understand and generate human language. LLMs predict the most likely next piece of text, one token at a time, which lets them answer questions, write and summarize documents, generate code, and power AI agents that take actions.

## GateWatch: Low-Overhead LLM Monitoring with Boolean Circuits

DevFeed: [GateWatch: Low-Overhead LLM Monitoring with Boolean Circuits](<https://devfeed.tech/articles/gatewatch-low-overhead-llm-monitoring-with-boolean-circuits-142814.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24619>)

Author: Firas Spanioli, Yaniv David

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [LLM security](<https://devfeed.tech/topics/llm-security.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [boolean](<https://devfeed.tech/tags/boolean.md>), [circuits](<https://devfeed.tech/tags/circuits.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-monitoring](<https://devfeed.tech/tags/llm-monitoring.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [overhead](<https://devfeed.tech/tags/overhead.md>)

### AI overview

GateWatch represents each monitoring probe as Boolean logic over a few binarized sparse autoencoder (SAE) features. Its serving system computes only selected features, streams their binary values across layers, and evaluates classifiers in parallel with lookup tables. On 24 Gemma-2-9B tasks, it averages 80.49% test accuracy with 3.8 SAE features and 3.5 logic gates per probe--1.17 percentage points below full 16K-feature SAE probes--while matching or exceeding them on 10 tasks and reducing probe memory by 99.98%. In vLLM, 1,000 concurrent classifiers retain 97-98% of baseline throughput and 99.8% of KV-cache capacity.

### Source excerpt

arXiv:2611.24619v1 Announce Type: new Abstract: As large language models (LLMs) and LLM-based agents are increasingly deployed, monitoring their behavior becomes important for detecting unsafe or unintended actions. Activation probes offer an efficient approach, but existing methods face trade-offs between accuracy, interpretability, and computational overhead. Probes on raw activations lack interpretable inputs, whereas sparse autoencoder (SAE) probes use interpretable features but often rely on additive classifiers and large SAE encoders, limiting interpretability and increasing memory overhead. We introduce GateWatch, a compact, interpretable classifier that expresses its entire decision rule as an explicit Boolean logic over a small number of binarized SAE features. We explore three approaches to constructing these circuits under a specified gate budget. For efficient deployment, GateWatchServe computes only selected SAE features, enables streaming of their binary values across layers during inference, and evaluates multiple GateWatch classifiers in parallel using compact lookup tables. Across 24 tasks on Gemma-2-9B, GateWatch achieves 80.49% mean test accuracy using only 3.8 SAE features and 3.5 logic gates per probe on average, just 1.17 percentage points below full 16K-feature SAE probes, while matching or outperforming them on 10 tasks and reducing probe memory by 99.98%. Inside vLLM, 1,000 concurrent multi-layer, multi-token GateWatch classifiers retain 97-98% of baseline throughput and 99.8% of KV-cache capacity.

## A Survey on RLOps for Large Language Models

DevFeed: [A Survey on RLOps for Large Language Models](<https://devfeed.tech/articles/a-survey-on-rlops-for-large-language-models-142184.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23357>)

Author: Lingzhe Zhang, Tong Jia, Chiming Duan, Minghua He, Leyi Pan, Kening Zheng, Zhaoyang Liu, Bolin Ding, Philip S. Yu, Ying Li

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>)

Tags: [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [github-repository](<https://devfeed.tech/tags/github-repository.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [observability](<https://devfeed.tech/tags/observability.md>), [recursive-self-improvement](<https://devfeed.tech/tags/recursive-self-improvement.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [reinforcement-learning-training](<https://devfeed.tech/tags/reinforcement-learning-training.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [shared-infrastructure](<https://devfeed.tech/tags/shared-infrastructure.md>), [survey](<https://devfeed.tech/tags/survey.md>), [taxonomy](<https://devfeed.tech/tags/taxonomy.md>)

### AI overview

The survey organizes methods for reliable and efficient LLM reinforcement-learning operations across five lifecycle stages: Training Setup, Experience Collection, Reward Computation, Policy Optimization, and Model Evaluation. It frames RLOps as a response to distributed-execution, feedback-computation, and learning-dynamics failures that can waste compute, slow training, or weaken improvements. For each stage, the survey examines operational concerns, management techniques, and evaluation evidence. It identifies future work in training observability, coordination across stages, generalizable RLOps, managing multiple runs, and supporting recursive self-improvement. The supplied text is an abstract and does not provide details on individual methods or comparative results.

### Source excerpt

arXiv:2611.23357v1 Announce Type: new Abstract: Reinforcement learning (RL) has become increasingly important for improving the reasoning, instruction-following, and task-solving capabilities of large language models (LLMs). However, scaling RL training exposes failures in distributed execution, feedback computation, and learning dynamics that can waste computational resources, slow training, and undermine model improvements. These challenges have motivated a growing body of techniques for LLM RLOps, which manages the efficiency and reliability of operations throughout the LLM reinforcement learning training lifecycle. Despite growing interest, these techniques remain scattered across research areas, with limited synthesis of their operational roles and interactions. To bridge this gap, we present a systematic survey of LLM RLOps methods. We organize existing approaches into five stages: Training Setup, Experience Collection, Reward Computation, Policy Optimization, and Model Evaluation. Within each stage, we examine operational concerns, management techniques, and evaluation evidence. Building on this taxonomy, we identify open challenges and future directions in training process observability, cross-stage coordination, generalizable RLOps, multi-run management, and RLOps for recursive self-improvement. These directions extend operational management from individual training runs to diverse training settings, shared infrastructure, and successive improvement cycles. We maintain a GitHub repository indexing relevant papers and open resources at https://github.com/zhanglingzhe0820/Awesome-RLOps.

## Training Language Models for Introspective Oversight

DevFeed: [Training Language Models for Introspective Oversight](<https://devfeed.tech/articles/training-language-models-for-introspective-oversight-142648.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23365>)

Author: Alex Lin Wang, Linus Yifeng Tang, Dami Choi, Neil Chowdhury

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [Uncertainty quantification LLMs](<https://devfeed.tech/topics/uncertainty-quantification-llms.md>)

Tags: [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [deception](<https://devfeed.tech/tags/deception.md>), [evaluations](<https://devfeed.tech/tags/evaluations.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [observer](<https://devfeed.tech/tags/observer.md>), [oversight](<https://devfeed.tech/tags/oversight.md>), [training](<https://devfeed.tech/tags/training.md>), [transcript](<https://devfeed.tech/tags/transcript.md>)

### AI overview

Introspective oversight trains a separate third-person observer role to answer questions about an assistant, with answers rewarded for matching the assistant's verified behavior. The approach aims to address limits of asking the assistant to report its own misbehavior, transcript monitors' reliance on visible outputs, and activation decoders' lack of context and tendency to confabulate. On five held-out oversight evaluations, an observer trained on Qwen3.6-27B achieved mean AUROC 0.907, compared with 0.864 for honesty training, 0.828 for transcript monitoring, and 0.796 for activation oracles. Training the observer also made the assistant easier to monitor: every baseline method improved on average when used with the fine-tuned assistant. These results suggest the approach is tractable for overseeing unintended behavior.

### Source excerpt

arXiv:2611.23365v1 Announce Type: new Abstract: Language models increasingly exhibit behaviors that are difficult to detect in transcripts, such as reward hacking, deception, and sycophancy. Asking the assistant follow-up questions about its actions is unreliable, since models are generally not trained to report their own misbehavior. As a result, approaches to oversight typically involve querying external models. These face limitations: external transcript monitoring operates only on visible model outputs, and activation decoders lack full context and are prone to confabulation. We introduce introspective oversight, which trains a new role in the model to answer arbitrary questions about the assistant, rewarding answers that are verified against the assistant's actual behavior. We implement this by adding a separate, third-person observer role that answers these queries, which we can train solely for accuracy and separately from the assistant. On five held-out oversight evaluations, an observer trained on Qwen3.6-27B obtains a mean AUROC of 0.907, outperforming honesty training (0.864), transcript monitoring (0.828), and activation oracles (0.796). We further find that training an observer makes the assistant itself easier to monitor, with every baseline method improving on average when applied to the fine-tuned assistant. These results suggest that training models to answer verifiable questions about themselves is a tractable approach to overseeing unintended behavior.

## Elastic and HoneyHive join forces to advance AI observability and evaluation

DevFeed: [Elastic and HoneyHive join forces to advance AI observability and evaluation](<https://devfeed.tech/articles/elastic-and-honeyhive-join-forces-to-advance-ai-observability-and-evaluation-146502.md>)

Original publisher: [Read original article](<https://www.elastic.co/blog/elastic-and-honeyhive>)

Author: Santosh Krishnan

Published: 2026-11-19T00:00:00Z

Content type: news

Language: en

Sources: [Elastic Blog](<https://devfeed.tech/sources/elastic-blog-elasticsearch-kibana-and-elk-stack.md>)

Topics: [agent observability](<https://devfeed.tech/topics/agent-observability.md>), [observability](<https://devfeed.tech/topics/observability.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>)

Tags: [actions](<https://devfeed.tech/tags/actions.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agent-behavior](<https://devfeed.tech/tags/agent-behavior.md>), [agent-workflows](<https://devfeed.tech/tags/agent-workflows.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [ai-applications](<https://devfeed.tech/tags/ai-applications.md>), [ai-observability](<https://devfeed.tech/tags/ai-observability.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [announce](<https://devfeed.tech/tags/announce.md>), [application-monitoring](<https://devfeed.tech/tags/application-monitoring.md>), [applications](<https://devfeed.tech/tags/applications.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [capabilities](<https://devfeed.tech/tags/capabilities.md>), [data](<https://devfeed.tech/tags/data.md>), [elastic](<https://devfeed.tech/tags/elastic.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [observability](<https://devfeed.tech/tags/observability.md>), [reliable-ai-agents](<https://devfeed.tech/tags/reliable-ai-agents.md>)

### AI overview

Elastic has entered into an agreement to acquire HoneyHive, an enterprise AI observability and evaluation platform. HoneyHive traces complex agent workflows and supports offline evaluation and experimentation before deployment, plus ongoing production monitoring and evaluation to identify unexpected behavior and guide improvements. The announcement distinguishes this agent-focused view of behavior and outcomes from traditional monitoring of application performance. Elastic aims to integrate HoneyHive's tracing and evaluation with its search and analytics foundation, linking agent quality to broader operational context. Existing HoneyHive customers are intended to remain supported during integration; a consolidated product roadmap is expected in coming months.

### Source excerpt

Today, we're taking a meaningful step toward helping customers build more reliable AI agents and applications. We're excited to announce that Elastic has entered into an agreement to acquire HoneyHive.

## How Airbnb uses cost per task to operate coding agents on Amazon Bedrock

DevFeed: [How Airbnb uses cost per task to operate coding agents on Amazon Bedrock](<https://devfeed.tech/articles/how-airbnb-uses-cost-per-task-to-operate-coding-agents-on-amazon-bedrock-141280.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/machine-learning/how-airbnb-uses-cost-per-task-to-operate-coding-agents-on-amazon-bedrock/>)

Author: Alan Yao

Published: 2026-11-18T21:48:19Z

Content type: article

Language: en

Sources: [Artificial Intelligence](<https://devfeed.tech/sources/artificial-intelligence.md>)

Topics: [Amazon Bedrock](<https://devfeed.tech/topics/amazon-bedrock.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [cloud cost management](<https://devfeed.tech/topics/cloud-cost-management.md>), [Amazon CloudWatch](<https://devfeed.tech/topics/amazon-cloudwatch.md>), [agentic-coding](<https://devfeed.tech/topics/agentic-coding.md>), [AI-assisted coding](<https://devfeed.tech/topics/ai-assisted-coding.md>), [rate-limiting](<https://devfeed.tech/topics/rate-limiting.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [airbnb](<https://devfeed.tech/tags/airbnb.md>), [amazon-bedrock](<https://devfeed.tech/tags/amazon-bedrock.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [cost](<https://devfeed.tech/tags/cost.md>), [customer-solutions](<https://devfeed.tech/tags/customer-solutions.md>), [evals](<https://devfeed.tech/tags/evals.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [latency](<https://devfeed.tech/tags/latency.md>), [model-selection](<https://devfeed.tech/tags/model-selection.md>), [observability](<https://devfeed.tech/tags/observability.md>), [production](<https://devfeed.tech/tags/production.md>), [quotas](<https://devfeed.tech/tags/quotas.md>), [spending](<https://devfeed.tech/tags/spending.md>), [tasks](<https://devfeed.tech/tags/tasks.md>)

### AI overview

Airbnb combines task-based evaluations with Amazon Bedrock production metrics to select and revisit coding-agent models. It scores models and reasoning-effort settings on representative engineering tasks, pricing all uncached input, cache reads, cache writes, and output tokens across a task; these results guide defaults and effort ceilings. Production can diverge: one model scored better at similar eval cost, but its users later spent several times more because requests cost about three times as much and users sent about three times as many requests. A newer model later scored higher at roughly one sixth the task cost, and heavy users' spend fell to about a seventh with similar request volume. For operations, Airbnb tracks latency, token counts, derived output speed, throttles, and estimated quota usage. Quota admission reserves uncached input, cache writes, and max_tokens, so lowering an oversized max_tokens can improve admission headroom without changing final quota consumption for unchanged output. Gateway records, IAM tags, inference profiles, and invocation logs support aggregate spend attribution; logs require access controls and retention settings.

### Source excerpt

Airbnb runs evals on real engineering tasks and uses Amazon Bedrock observability to choose models for its coding agents and control production costs. This post shows how cost per task guides model selection, why production spending can differ from eval results, and how to monitor latency, manage quotas, and attribute spend.

## What Is Context Engineering? Giving AI Agents the Right Information

DevFeed: [What Is Context Engineering? Giving AI Agents the Right Information](<https://devfeed.tech/articles/what-is-context-engineering-giving-ai-agents-the-right-information-141931.md>)

Original publisher: [Read original article](<https://montecarlo.ai/blog-context-engineering>)

Author: Ruslan Sultanov

Published: 2026-11-18T18:46:57Z

Content type: article

Language: en

Sources: [Monte Carlo](<https://devfeed.tech/sources/monte-carlo.md>)

Topics: [AI Context Management](<https://devfeed.tech/topics/ai-context-management.md>), [context window](<https://devfeed.tech/topics/context-window.md>), [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>), [Prompt Engineering](<https://devfeed.tech/topics/prompt-engineering.md>), [AI-Ready Data](<https://devfeed.tech/topics/ai-ready-data.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [context](<https://devfeed.tech/topics/context.md>)

Tags: [agent-memory](<https://devfeed.tech/tags/agent-memory.md>), [agent-trust](<https://devfeed.tech/tags/agent-trust.md>), [agent-trust-glossary](<https://devfeed.tech/tags/agent-trust-glossary.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [context-engineering](<https://devfeed.tech/tags/context-engineering.md>), [context-engineering-for-ai-agents](<https://devfeed.tech/tags/context-engineering-for-ai-agents.md>), [context-engineering-vs-prompt-engineering](<https://devfeed.tech/tags/context-engineering-vs-prompt-engineering.md>), [context-engineering-vs-rag](<https://devfeed.tech/tags/context-engineering-vs-rag.md>), [context-window](<https://devfeed.tech/tags/context-window.md>), [context-window-management](<https://devfeed.tech/tags/context-window-management.md>), [data](<https://devfeed.tech/tags/data.md>), [data-pipeline](<https://devfeed.tech/tags/data-pipeline.md>), [data-quality](<https://devfeed.tech/tags/data-quality.md>), [glossary](<https://devfeed.tech/tags/glossary.md>), [llm-context](<https://devfeed.tech/tags/llm-context.md>), [prompts](<https://devfeed.tech/tags/prompts.md>), [what-is-context-engineering](<https://devfeed.tech/tags/what-is-context-engineering.md>)

### AI overview

Context engineering shapes everything an agent receives--instructions, retrieved data, tool results, and memory--so it has a small, relevant set of information at each step. Prompt engineering focuses on instruction wording; RAG fetches documents for the prompt; context engineering encompasses both, along with tool design, memory, and what to omit. Techniques include just-in-time retrieval, reranking and filtering, compressing long outputs, using sub-agents with separate context, storing notes externally, and structuring inputs. These cannot correct stale or faulty source data. The excerpt recommends monitoring freshness and volume, tracing outputs to source documents and versions, recording retrievals, and evaluating groundedness and correctness. It notes that larger context windows do not remove the need to select and manage information.

### Source excerpt

Two teams build the same support agent on the same model with nearly identical prompts. One answers correctly nine times out of ten. The other confidently quotes a shipping policy that changed in the spring. The difference isn't the model or the wording. It's what each agent was given to read before it answered. That's ...

## What Is LLM Monitoring? What to Track Once Your Model Is Live

DevFeed: [What Is LLM Monitoring? What to Track Once Your Model Is Live](<https://devfeed.tech/articles/what-is-llm-monitoring-what-to-track-once-your-model-is-live-141933.md>)

Original publisher: [Read original article](<https://montecarlo.ai/blog-llm-monitoring>)

Author: Ruslan Sultanov

Published: 2026-11-18T18:07:29Z

Content type: article

Language: en

Sources: [Monte Carlo](<https://devfeed.tech/sources/monte-carlo.md>)

Topics: [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [Monitoring](<https://devfeed.tech/topics/monitoring.md>), [ai observability](<https://devfeed.tech/topics/ai-observability.md>), [Monitoring & Alerting](<https://devfeed.tech/topics/monitoring-alerting.md>), [data observability](<https://devfeed.tech/topics/data-observability.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [agent-trust](<https://devfeed.tech/tags/agent-trust.md>), [agent-trust-glossary](<https://devfeed.tech/tags/agent-trust-glossary.md>), [glossary](<https://devfeed.tech/tags/glossary.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-monitoring](<https://devfeed.tech/tags/llm-monitoring.md>), [llm-monitoring-metrics](<https://devfeed.tech/tags/llm-monitoring-metrics.md>), [llm-monitoring-vs-observability](<https://devfeed.tech/tags/llm-monitoring-vs-observability.md>), [llm-performance-monitoring](<https://devfeed.tech/tags/llm-performance-monitoring.md>), [llm-quality-monitoring](<https://devfeed.tech/tags/llm-quality-monitoring.md>), [model](<https://devfeed.tech/tags/model.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [monitoring-llms-in-production](<https://devfeed.tech/tags/monitoring-llms-in-production.md>), [what-is-llm-monitoring](<https://devfeed.tech/tags/what-is-llm-monitoring.md>)

### AI overview

LLM monitoring catches production failures that uptime, latency, and error dashboards miss: an app can be fast and available while returning incorrect or unsafe answers. It continuously tracks performance (latency, tokens, cost, errors), context (source freshness and retrieval), behavior (tool calls, retries, refusals), and output quality (groundedness, relevance, correctness, safety, and user feedback). A practical setup traces requests across retrieval, model, and tool calls; scores traffic with deterministic checks and LLM-as-judge evaluations; alerts on trends or anomalies; then uses trace context to investigate causes such as stale data, prompt changes, or model updates. Evals assess fixed examples before release, monitoring detects production changes, and observability supplies traces and lineage to explain them. The article recommends versioning traces, establishing baselines, checking judges against human review, and assigning alert owners.

### Source excerpt

Your LLM app has been live for a month. Latency is under a second, the error rate is zero, and uptime is perfect. Meanwhile, a product manager forwards a screenshot: the bot told a customer about a discount that ended in March. Nothing on the dashboard moved. That's the gap LLM monitoring fills. This guide ...

## LMLineage: Content Lineage and Context Governance for LLM Agents

DevFeed: [LMLineage: Content Lineage and Context Governance for LLM Agents](<https://devfeed.tech/articles/lmlineage-content-lineage-and-context-governance-for-llm-agents-140041.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22511>)

Author: Wei Hou, Jingxin Geng, Xin Chen, Zheqing Jiang

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [AI Context Management](<https://devfeed.tech/topics/ai-context-management.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [application](<https://devfeed.tech/tags/application.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [attribution](<https://devfeed.tech/tags/attribution.md>), [band](<https://devfeed.tech/tags/band.md>), [bias](<https://devfeed.tech/tags/bias.md>), [bounded](<https://devfeed.tech/tags/bounded.md>), [build](<https://devfeed.tech/tags/build.md>), [case](<https://devfeed.tech/tags/case.md>), [case-study](<https://devfeed.tech/tags/case-study.md>), [compaction](<https://devfeed.tech/tags/compaction.md>), [content](<https://devfeed.tech/tags/content.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [governance](<https://devfeed.tech/tags/governance.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-agents](<https://devfeed.tech/tags/llm-agents.md>)

### AI overview

LMLineage attributes each agent output sentence to supporting input spans with calibrated confidence, using an out-of-band process that needs neither model white-box access nor extra inference passes. Its context-governance layer detects input documents that are no longer referenced and compacts those selectively while preserving user instructions verbatim. Across a small corpus of 71 annotated sentences and 90 multi-turn session turns using three generator models, window-level embedding similarity aligned with paraphrases better than whole-document or lexical matching. A greedy-window fast path was faster and more accurate on average than exhaustive line matching, whose multiple-comparisons bias raised web recall at the cost of overall accuracy. Attribution took about one second per turn on one GPU and remained stable through 40 turns. Drift detection had 0% false compaction in the reported evaluation, with recall limited by attribution accuracy.

### Source excerpt

arXiv:2611.22511v1 Announce Type: new Abstract: Interpretability for AI agents should not be confined to understanding the model's internal weights. It must also cover the agent's reasoning and decision-making process over multi-source context. We present LMLineage, an out-of-band system that attributes every output sentence of an agent session to its supporting input spans with calibrated confidences, without requiring white-box access to the model or any additional inference passes. On top of the lineage metadata, we build concept-drift micro compaction: instead of waiting for the context-length limit and summarizing everything, the system detects when previously used input documents have stopped being referenced ("drifted") and surgically compacts only those, preserving user instructions verbatim. We evaluate on a multi-tier corpus: 71 gold annotated sentences from 10 Chinese web-QA cases, plus 90 turns of independently collected multi-turn sessions across three generator models (gpt-4o-mini, qwen3.8-27b, deepseek-v4-flash). Results show that (i) window-level embedding similarity is far better aligned with LLM paraphrasing than whole document or lexical matching; (ii) the greedy-window fast path is both faster and more accurate on average than exhaustive line-matching (a multiple-comparisons bias in the latter inflates web recall at the cost of overall accuracy); (iii) per-turn attribution completes in 1s on a single GPU, remaining stable as context accumulates to 40 turns; and (iv) concept-drift detection is conservative (0% false compaction) with recall bounded by attribution accuracy. We further describe the concept-drift micro compaction design and an interactive incident-tracing/memory-cleaning case study. A companion patent application covers the lineage construction method.

## Context Has a Price: Adding and Removing Context for LLM Coding Agents under Prefix Caching

DevFeed: [Context Has a Price: Adding and Removing Context for LLM Coding Agents under Prefix Caching](<https://devfeed.tech/articles/context-has-a-price-adding-and-removing-context-for-llm-coding-agents-under-prefix-caching-139987.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22238>)

Author: Zerui Yang, Miaomiao Dong, Quan Xue, Chaorui Zhang, Xuanyu Zheng, Xiyuan Wang, Haolin Ruan, Xueyan Niu, Guangcheng Song, Weixi Zhang, Wei Han, Bo Bai

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [AI Context Management](<https://devfeed.tech/topics/ai-context-management.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [reproducibility](<https://devfeed.tech/topics/reproducibility.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [caching](<https://devfeed.tech/tags/caching.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [concurrent](<https://devfeed.tech/tags/concurrent.md>), [context](<https://devfeed.tech/tags/context.md>), [context-management](<https://devfeed.tech/tags/context-management.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [llm](<https://devfeed.tech/tags/llm.md>), [measurement](<https://devfeed.tech/tags/measurement.md>), [prefix-caching](<https://devfeed.tech/tags/prefix-caching.md>), [prompts](<https://devfeed.tech/tags/prompts.md>), [swe-bench](<https://devfeed.tech/tags/swe-bench.md>), [token](<https://devfeed.tech/tags/token.md>), [tokens](<https://devfeed.tech/tags/tokens.md>)

### AI overview

Across more than 20,000 runs of open models on SWE-bench Verified, SWE-bench and Loc-Bench, the study finds that context changes can alter coding-agent costs in opposite ways depending on prefix-cache pricing and subsequent behavior. In the main evidence-package experiments, retrieved code cut read and search commands by 18-34% but raised cost by 33-42% when cached input cost one tenth as much as uncached input, with no measurable resolution improvement. Removing old observations saved 19-36% without a cache discount but raised cost by 38-116% under the tenth-price cache. Shorter prompts did not guarantee cheaper runs because agents sometimes took more steps. Concurrent experiments could share cached prefixes and distort comparisons. The authors therefore evaluate context-management methods by accounting separately for uncached input, cached input and output tokens, and by measuring agents' resulting behavior.

### Source excerpt

arXiv:2611.22238v1 Announce Type: new Abstract: Coding agents repeatedly send their conversation history to a language model. Adding useful code may reduce search, while removing old tool outputs may shorten later prompts. But do these changes reduce the cost of solving a task? We study this as a measurement problem in more than 20,000 runs of open models on SWE-bench Verified, SWE-bench and Loc-Bench. We account separately for uncached input, cached input and output tokens. Three findings emerge. First, in our main evidence-package experiments, retrieved code reduces read and search commands by 18--34%, but increases cost by 33--42% when cached input costs one tenth of uncached input, without a measurable improvement in resolution. Second, removing old observations can either save money or increase cost: the same approach saves 19--36% without a cache discount but increases cost by 38--116% with a tenth-price cache. Third, shorter prompts do not guarantee cheaper runs because agents may respond by taking more steps. We also find that concurrent experiments can share cached prefixes and distort cost comparisons. These results show why context-management methods should be evaluated using both token prices and the agent's subsequent behavior.

## How we fine-tuned a faster model for Patterns on Baseten

DevFeed: [How we fine-tuned a faster model for Patterns on Baseten](<https://devfeed.tech/articles/how-we-fine-tuned-a-faster-model-for-patterns-on-baseten-142009.md>)

Original publisher: [Read original article](<https://www.datadoghq.com/blog/fine-tuning-patterns-baseten/>)

Author: Fouad Wahabi; Vincent Cornet; Alex Barksdale

Published: 2026-11-18T00:00:00Z

Content type: article

Language: en

Sources: [Datadog](<https://devfeed.tech/sources/datadog-the-monitor-blog.md>)

Topics: [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [ai observability](<https://devfeed.tech/topics/ai-observability.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Enhancing AI model performance](<https://devfeed.tech/topics/enhancing-ai-model-performance.md>), [baseten](<https://devfeed.tech/topics/baseten.md>), [lora](<https://devfeed.tech/topics/lora.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [model-deployment](<https://devfeed.tech/topics/model-deployment.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [3](<https://devfeed.tech/tags/3.md>), [78](<https://devfeed.tech/tags/78.md>), [about](<https://devfeed.tech/tags/about.md>), [actions](<https://devfeed.tech/tags/actions.md>), [adapter](<https://devfeed.tech/tags/adapter.md>), [after](<https://devfeed.tech/tags/after.md>), [agent-observability](<https://devfeed.tech/tags/agent-observability.md>), [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [baseten](<https://devfeed.tech/tags/baseten.md>), [datadog](<https://devfeed.tech/tags/datadog.md>), [faster](<https://devfeed.tech/tags/faster.md>), [fine](<https://devfeed.tech/tags/fine.md>), [generation](<https://devfeed.tech/tags/generation.md>), [model](<https://devfeed.tech/tags/model.md>), [one](<https://devfeed.tech/tags/one.md>), [patterns](<https://devfeed.tech/tags/patterns.md>), [production](<https://devfeed.tech/tags/production.md>), [time](<https://devfeed.tech/tags/time.md>), [traces](<https://devfeed.tech/tags/traces.md>), [under](<https://devfeed.tech/tags/under.md>), [using](<https://devfeed.tech/tags/using.md>)

### AI overview

Datadog fine-tuned Qwen 3.5 9B for Patterns' repetitive interaction summarization, reducing average generation time on 150 held-out cases from 44.7 seconds to 0.89 seconds. Patterns' production LLM spans supplied examples pairing interactions and clustering instructions with existing summaries; rows were split by trace to limit test leakage. On Baseten, the base model often exhausted its token budget before producing a parseable answer: only 48% of its outputs finished, and 44.7% passed the LLM judge. After LoRA fine-tuning with early stopping based on validation loss, all 150 outputs finished and 93.3% passed. Datadog's pup finetune workflow supported dataset review, benchmarking, training, and evaluation, with approval before paid steps. The reported evaluation used five held-out traces and suggests the model generalized across the tested clustering instructions.

### Source excerpt

Datadog fine-tuned a summarization model using production traces, cutting average generation time from about 45 seconds to under one second.

## Engineering AI Discover

DevFeed: [Engineering AI Discover](<https://devfeed.tech/articles/engineering-ai-discover-139631.md>)

Original publisher: [Read original article](<https://netflixtechblog.com/engineering-ai-discover-e244944cdc38?source=rss----2615bd06b42e---4>)

Author: Netflix Technology Blog

Published: 2026-11-17T17:01:02Z

Content type: article

Language: en

Sources: [Netflix](<https://devfeed.tech/sources/netflix-techblog-medium.md>)

Topics: [recommendation systems](<https://devfeed.tech/topics/recommendation-systems.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [ReAct agent Python](<https://devfeed.tech/topics/react-agent-python.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [AI Context Management](<https://devfeed.tech/topics/ai-context-management.md>), [Production Engineering](<https://devfeed.tech/topics/production-engineering.md>), [LLM security](<https://devfeed.tech/topics/llm-security.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agent-harness](<https://devfeed.tech/tags/agent-harness.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [app](<https://devfeed.tech/tags/app.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [challenges](<https://devfeed.tech/tags/challenges.md>), [code](<https://devfeed.tech/tags/code.md>), [discovery](<https://devfeed.tech/tags/discovery.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [engineering-ai](<https://devfeed.tech/tags/engineering-ai.md>), [generative](<https://devfeed.tech/tags/generative.md>), [generative-ai](<https://devfeed.tech/tags/generative-ai.md>), [harness-engineering](<https://devfeed.tech/tags/harness-engineering.md>), [layer](<https://devfeed.tech/tags/layer.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llms](<https://devfeed.tech/tags/llms.md>), [recommendation](<https://devfeed.tech/tags/recommendation.md>), [recommendation-system](<https://devfeed.tech/tags/recommendation-system.md>), [recommendation-systems](<https://devfeed.tech/tags/recommendation-systems.md>), [recommendations](<https://devfeed.tech/tags/recommendations.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [rules](<https://devfeed.tech/tags/rules.md>), [systems](<https://devfeed.tech/tags/systems.md>), [tv](<https://devfeed.tech/tags/tv.md>)

### AI overview

AI Discover uses a controlled agent harness to make TV recommendations steerable while keeping results grounded, safe and responsive. A controller selects an established plan or validates an LLM-drafted plan; an executor runs approved capabilities, passing typed outputs between steps and using parallel execution, deadlines and fallbacks. Retrieval and ranking systems supply real catalog titles, while LLMs handle selected tasks such as generating themes and explanations. Planning before execution bounds the work and allows slow or invalid plans to fall back to tested ones; asynchronous serving acknowledges a remote press immediately and returns the generated page when ready. Rules and access limits are enforced in code, with structured request records supporting offline evaluation and replay. Model and capability choices can be changed through server-side configuration. Members steer recommendations with selectable themes and tuning options, whose preferences are stored alongside precomputed taste summaries; the current experience has no free-text input or open-ended chat. The authors say production cost, latency, member wait tolerance and multi-turn evaluation remain areas to study

### Source excerpt

The challenges and decisions behind AI Discover's intelligence layer By Varun Khaitan, Peng Ren, Gary Tang, Itay Teller, Atif Faheem, Nipun Kumar, Donnie DeBoer, Eduardo Lundgren, Matteo Rinaldi, Raghav Bakshi We recently introduced AI Discover, a new tab in our TV app where a small group of members can try new ways to find something to watch. For years, our recommendations have worked in the background, inferring what a member might enjoy from their viewing history. AI Discover moves from that silent ranking toward steerable discovery. A member picks a theme, say "Quick Watches," and gets a row of matching titles, a short note on why they fit and a few narrower themes to try next. From there, they can keep narrowing, head somewhere new or tune what they see. Building AI Discover raised a question many teams will recognize: how do you deliver a better member experience with generative AI on TV without sacrificing speed, reliability or safety? Our answer is a new intelligence layer, the agent harness behind AI Discover. For each request, it decides how to combine LLMs with our recommendation systems. It builds on years of work understanding what members enjoy, and on our recent work applying LLMs to recommendations and generative experiences. The layer itself plans each request, runs it across those systems, serves the result and evaluates it offline. This post covers the challenges we designed for and the key decisions we made along the way. Why an intelligence layer? Every Netflix page combines recommendations with rules. We pick titles a member may enjoy, and eligibility, maturity and safety rules decide what can appear. An experience built on LLMs needs the same rules, plus code that decides when to call an LLM and how to handle its output. That code is called an agent harness, and it's the core of our intelligence layer. The simplest implementation hands the entire request to an LLM and lets it pick the titles, write the labels and decide what happens next. Howe

## Datadog Brings Production Experimentation to AI Agent Observability

DevFeed: [Datadog Brings Production Experimentation to AI Agent Observability](<https://devfeed.tech/articles/datadog-brings-production-experimentation-to-ai-agent-observability-137152.md>)

Original publisher: [Read original article](<https://www.infoq.com/news/2026/11/datadog-ai-agent-observability/>)

Author: Craig Risi

Published: 2026-11-17T12:00:00Z

Content type: article

Language: en

Sources: [InfoQ](<https://devfeed.tech/sources/infoq.md>)

Topics: [experiments](<https://devfeed.tech/topics/experiments.md>), [agent observability](<https://devfeed.tech/topics/agent-observability.md>), [A/B Testing](<https://devfeed.tech/topics/a-b-testing.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [observability](<https://devfeed.tech/topics/observability.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agent-observability](<https://devfeed.tech/tags/agent-observability.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [ai-agent-observability](<https://devfeed.tech/tags/ai-agent-observability.md>), [ai-coding-agents](<https://devfeed.tech/tags/ai-coding-agents.md>), [ai-ml-data-engineering](<https://devfeed.tech/tags/ai-ml-data-engineering.md>), [aiops](<https://devfeed.tech/tags/aiops.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [datadog](<https://devfeed.tech/tags/datadog.md>), [datadog-ai-agent-observability](<https://devfeed.tech/tags/datadog-ai-agent-observability.md>), [devops](<https://devfeed.tech/tags/devops.md>), [experimentation](<https://devfeed.tech/tags/experimentation.md>), [ml-data-engineering](<https://devfeed.tech/tags/ml-data-engineering.md>), [news](<https://devfeed.tech/tags/news.md>), [observability](<https://devfeed.tech/tags/observability.md>), [production](<https://devfeed.tech/tags/production.md>)

### AI overview

Datadog's Online Experiments lets teams compare AI agent versions on randomized real production traffic, pairing evaluator scores for response quality, accuracy, and completeness with latency, token use, estimated cost, and errors. Feature Flags support small, staged rollouts, while linked traces expose tool calls and results so teams can investigate differences and turn failures into offline regression tests. In the article's example, GPT-5.6 Luna and GPT-6 Luna received an even traffic split; their quality scores were close (3.84 and 3.86), and the newer model's estimated cost was 44.3% lower. Its execution duration and workflow errors appeared higher, but confidence intervals were too broad to establish regressions. Traces also revealed more invalid tool parameters, often recovered through retries, prompting tool-handling improvements before rollout. The approach complements offline evaluations, whose findings depend on how representative their datasets and scoring criteria are.

### Source excerpt

Datadog is extending its Agent Observability platform with Online Experiments, allowing engineering teams to compare AI agent versions against real production traffic rather than relying exclusively on predefined evaluation datasets. By Craig Risi

## The Silent Cost of Conversation History in LLM Reasoning

DevFeed: [The Silent Cost of Conversation History in LLM Reasoning](<https://devfeed.tech/articles/the-silent-cost-of-conversation-history-in-llm-reasoning-137532.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20071>)

Author: Ali \\c{S}enol

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [AI Context Management](<https://devfeed.tech/topics/ai-context-management.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>)

Tags: [cdr](<https://devfeed.tech/tags/cdr.md>), [coherence](<https://devfeed.tech/tags/coherence.md>), [consistency](<https://devfeed.tech/tags/consistency.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [early-warning](<https://devfeed.tech/tags/early-warning.md>), [gpt](<https://devfeed.tech/tags/gpt.md>), [history](<https://devfeed.tech/tags/history.md>), [information-systems](<https://devfeed.tech/tags/information-systems.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [llama](<https://devfeed.tech/tags/llama.md>), [llm-reasoning](<https://devfeed.tech/tags/llm-reasoning.md>), [llms](<https://devfeed.tech/tags/llms.md>), [math](<https://devfeed.tech/tags/math.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [mistral](<https://devfeed.tech/tags/mistral.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [monitor](<https://devfeed.tech/tags/monitor.md>), [monitors](<https://devfeed.tech/tags/monitors.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>)

### AI overview

Retaining earlier replies in the conversation reduced final success on 48 sharded math tasks by 0.21 for GPT-4o-mini and 0.29 for Mistral-7B; Llama-4 Scout showed a nonsignificant decline. In 100-turn state-tracking dialogues, full-history accuracy fell from about 0.6 to 0.1 within twenty turns at matched difficulty, while solver-verified state summaries kept accuracy 0.52-0.59 higher. Consistency and logical coherence declined much less than accuracy, and stability increased. The tested process metrics and drift detectors did not provide robust, model-general early warning; repeat-based metrics were complete only for GPT-4o-mini, and single-stream monitors detected only very large drops. The authors recommend storing task state outside the transcript and monitoring across sessions.

### Source excerpt

arXiv:2611.20071v1 Announce Type: cross Abstract: Large language models (LLMs) in multi-turn information systems accumulate their own earlier replies and a growing task state. We asked whether this conversation history lowers accuracy, and whether process-level quality metrics warn before it does. Each analysis was fixed before its outcomes were computed. In Study 1, on 48 sharded math tasks, retaining its replies lowered final success by 0.21 for GPT-4o-mini and 0.29 for Mistral-7B (Holm-adjusted p = 0.026 and 0.004); Llama-4 Scout showed the same direction, not significant. Study 2 introduces CDR-Stream, 100-turn state-tracking dialogues with an exact answer per turn and per-model difficulty calibration. With the full history, accuracy fell from about 0.6 to 0.1 within twenty turns at matched difficulty, whereas a solver-verified state summary kept it 0.52-0.59 higher (Holm-adjusted p less than 0.001). Consistency and logical coherence fell far less than accuracy, and stability rose. None of these metrics, a paired readiness measure or calibrated single-stream drift detectors gave robust, model-general early warning, although in Study 2 the repeat-based metrics were complete only for GPT-4o-mini. In the monitors we tested, a single 100-turn stream had power only for very large drops. Systems built on LLMs should keep task state outside the transcript and monitor across sessions.

## When the Cheapest Feasible Router Fails: Auditing Deployment Commitments in Cost-Aware LLM Routing

DevFeed: [When the Cheapest Feasible Router Fails: Auditing Deployment Commitments in Cost-Aware LLM Routing](<https://devfeed.tech/articles/when-the-cheapest-feasible-router-fails-auditing-deployment-commitments-in-cost-aware-llm-routing-139046.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20417>)

Author: Minxuan Hu, Qishi Zhan, Ziheng Chen, Zhengtao Yao, Zihan Dong, Jiayu Yi, Wenxi Sun

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM security](<https://devfeed.tech/topics/llm-security.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [decision-making](<https://devfeed.tech/topics/decision-making.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [audit](<https://devfeed.tech/topics/audit.md>)

Tags: [22](<https://devfeed.tech/tags/22.md>), [auditing](<https://devfeed.tech/tags/auditing.md>), [control](<https://devfeed.tech/tags/control.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [failure](<https://devfeed.tech/tags/failure.md>), [inference](<https://devfeed.tech/tags/inference.md>), [llm](<https://devfeed.tech/tags/llm.md>), [nested](<https://devfeed.tech/tags/nested.md>), [operational](<https://devfeed.tech/tags/operational.md>), [policies](<https://devfeed.tech/tags/policies.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [risk](<https://devfeed.tech/tags/risk.md>), [router](<https://devfeed.tech/tags/router.md>), [routing](<https://devfeed.tech/tags/routing.md>), [semantic](<https://devfeed.tech/tags/semantic.md>), [service](<https://devfeed.tech/tags/service.md>), [serving](<https://devfeed.tech/tags/serving.md>)

### AI overview

Across 19 RouterBench task families, the cheapest router that passed an empirical audit violated the held-out service contract in 32.22% of its deployments, despite deploying in 91.82% of opportunities. The study treats deployment as a separate commitment decision using disjoint construction, audit, and test samples. Its Menu-Aware Routing Commitment (MARC) uses audit evidence and a serving-cost tolerance to keep the apparent winner, choose a supported alternative, or defer. The policywise variant MARC-P reduced observed conditional failure to 0.73% while deploying in 85.79% of opportunities, at higher serving cost. Fixed-margin and formal risk-control procedures were competitive alternatives. Nested audits and learned semantic routing showed related effects. These findings are reported in the supplied abstract.

### Source excerpt

arXiv:2611.20417v1 Announce Type: new Abstract: Cost-aware LLM routing often selects the cheapest policy meeting empirical risk and coverage constraints. Yet a router that passes an audit can violate its fixed service contract on new requests, even when candidate policies are constructed independently. We study this deployment-commitment decision with disjoint construction, audit, and test samples. Across 19 RouterBench task families, the cheapest audit-feasible router deploys in 91.82% of opportunities but violates the held-out contract in 32.22% of deployments. We introduce Menu-Aware Routing Commitment (MARC), which qualifies policies using audit evidence and a serving-cost tolerance to retain the apparent winner, deploy a supported alternative, or defer. Its practical policywise variant (MARC-P) reduces observed conditional failure to 0.73% at 85.79% deployment, with higher serving cost; fixed-margin and formal risk-control procedures provide competitive alternatives. Nested audits show how added evidence changes deployable policy sets, and learned semantic routing exhibits the same commitment failure. These results establish deployment commitment as a distinct operational decision connecting router evaluation, service reliability, and inference cost.

## Internal Verdicts Track Evidence: A Template-Controlled Neutrosophic Reading of Epistemic States in Large Language Models via the Jacobian Lens

DevFeed: [Internal Verdicts Track Evidence: A Template-Controlled Neutrosophic Reading of Epistemic States in Large Language Models via the Jacobian Lens](<https://devfeed.tech/articles/internal-verdicts-track-evidence-a-template-controlled-neutrosophic-reading-of-epistemic-states-in-large-language-models-via-the-jacobian-lens-138749.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18602>)

Author: Maikel Yelandi Leyva-Vazquez, Alexis Matheu Perez, Florentin Smarandache

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [qwen](<https://devfeed.tech/topics/qwen.md>), [audit](<https://devfeed.tech/topics/audit.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>)

Tags: [auditing](<https://devfeed.tech/tags/auditing.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [inside](<https://devfeed.tech/tags/inside.md>), [internals](<https://devfeed.tech/tags/internals.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [lens](<https://devfeed.tech/tags/lens.md>), [measurement](<https://devfeed.tech/tags/measurement.md>), [models](<https://devfeed.tech/tags/models.md>), [qwen3-5-4b](<https://devfeed.tech/tags/qwen3-5-4b.md>), [template](<https://devfeed.tech/tags/template.md>), [templates](<https://devfeed.tech/tags/templates.md>)

### AI overview

Using the Jacobian lens on Qwen3.5-4B, the authors read layer-wise truth, indeterminacy, and falsity signals from support, refutation, and hedging lexicons. Initial patterns under conflicting evidence and false belief largely disappeared under matched prompt templates, indicating sensitivity to prompt grammar. A verdict balance, B = log10(F) − log10(T), remained associated with evidence polarity under identical templates: roughly three orders of magnitude of separation across all nine reported layers, with Cohen's d of 2.3-2.5. It also residually distinguished false from true first-person beliefs. The abstract presents this two-battery comparison as a reusable protocol for auditing lens-based readings of model internals.

### Source excerpt

arXiv:2611.18602v1 Announce Type: cross Abstract: For twenty-five years, neutrosophic logic has assigned independent degrees of truth (T), indeterminacy (I), and falsity (F) to epistemic states that classical and fuzzy semantics cannot express. Previous applications measured these components on outputs: answers, judgments, expert evaluations. This paper reports, to our knowledge, the first neutrosophic reading of components inside the internal representations of a large language model. Using the recently released Jacobian lens on Qwen3.5-4B, we project intermediate-layer readouts onto lexicons of support, refutation, and hedging, obtaining layer-wise (T, I, F) profiles under conflicting evidence, first-person false belief, and factual control (n = 20 each). A first battery yielded three striking signatures: a surge of refutation mass under conflict, sustained indeterminacy under false belief, and an apparent "verdict collapse" before the output layer. A second, template-controlled battery showed that all three, as initially stated, were largely artifacts of prompt grammar: matched templates with agreeing evidence inflate the same lexicon masses, true beliefs elicit the same hedging, and the model verbalizes its verdict when allowed to generate. What survives the controls is stronger than what died: the neutrosophic verdict balance B = log10(F) - log10(T) tracks evidence polarity under identical templates (about 3 orders of magnitude of separation, Cohen's d = 2.3-2.5, p < 10^-6 in 9 of 9 layers; paired Wilcoxon: median difference +2.4 orders, 19/20 items, p = 3.8x10^-6) and residually discriminates false from true first-person beliefs. We argue that the independence of the neutrosophic components is what keeps the common-mode channel observable, making both the measurement and its self-correction possible, and we distill the two-battery design into a reusable protocol for auditing language model internals with lens-based readouts.

## From Voice AI Demo to Production: Why Infrastructure and Agent Supervision Matter

DevFeed: [From Voice AI Demo to Production: Why Infrastructure and Agent Supervision Matter](<https://devfeed.tech/articles/from-voice-ai-demo-to-production-why-infrastructure-and-agent-supervision-matter-140356.md>)

Original publisher: [Read original article](<https://www.twilio.com/en-us/blog/products/new-capabilities-conversation-relay-intelligence-twilio-agent-connect>)

Author: Lenore Files

Published: 2026-11-17T00:00:00Z

Content type: release

Language: en

Sources: [Twilio](<https://devfeed.tech/sources/twilio-blog.md>), [Twilio Blog](<https://devfeed.tech/sources/twilio-blog-2.md>)

Topics: [Conversational AI](<https://devfeed.tech/topics/conversational-ai.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [AI Integration Strategies](<https://devfeed.tech/topics/ai-integration-strategies.md>), [agent observability](<https://devfeed.tech/topics/agent-observability.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [observability](<https://devfeed.tech/topics/observability.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agent-behavior](<https://devfeed.tech/tags/agent-behavior.md>), [agent-frameworks](<https://devfeed.tech/tags/agent-frameworks.md>), [agent-runtime](<https://devfeed.tech/tags/agent-runtime.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [ai-infrastructure](<https://devfeed.tech/tags/ai-infrastructure.md>), [ai-powered](<https://devfeed.tech/tags/ai-powered.md>), [ai-stack](<https://devfeed.tech/tags/ai-stack.md>), [connect](<https://devfeed.tech/tags/connect.md>), [conversation-intelligence](<https://devfeed.tech/tags/conversation-intelligence.md>), [conversation-relay](<https://devfeed.tech/tags/conversation-relay.md>), [demo](<https://devfeed.tech/tags/demo.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [product-news](<https://devfeed.tech/tags/product-news.md>), [production](<https://devfeed.tech/tags/production.md>), [supervision](<https://devfeed.tech/tags/supervision.md>), [voice](<https://devfeed.tech/tags/voice.md>), [voice-ai](<https://devfeed.tech/tags/voice-ai.md>)

### AI overview

Twilio presents production Voice AI as a combination of real-time communications reliability, flexibility across AI providers, and supervision of actual customer conversations. Conversation Relay adds Smart Turn Detection to reduce premature replies, Automated STT Failover to switch to a backup provider during latency, rate limits, or stream disruptions, mid-call updates to speech hints, language, and voice settings, HD Audio, and support for ElevenLabs v4 Turbo. Twilio Agent Connect connects external agent frameworks and LLM runtimes to Twilio voice and messaging, letting teams keep their chosen models and business logic while Twilio supplies the communications layer. Conversation Intelligence offers transcript-first review without agent-code changes or intrusive SDKs; its public-beta Twilio-authored operators assess signals such as task completion and hallucinations, while its GA On-Demand Execution API runs Language Operators on demand alongside automated triggers. The article positions these features as tools for continuity, provider choice, and quality monitoring at scale, not as Twilio-provided agents or models.

### Source excerpt

Explore Twilio's new Voice AI infrastructure features from Conversation Relay, Agent Connect, and Conversation Intelligence.

## Monitor your Azure AI stack and get broad Azure coverage with Datadog

DevFeed: [Monitor your Azure AI stack and get broad Azure coverage with Datadog](<https://devfeed.tech/articles/monitor-your-azure-ai-stack-and-get-broad-azure-coverage-with-datadog-139518.md>)

Original publisher: [Read original article](<https://www.datadoghq.com/blog/azure-ai-monitoring-ignite-2026/>)

Author: Cody Murray-Bruce

Published: 2026-11-17T00:00:00Z

Content type: article

Language: en

Sources: [Datadog](<https://devfeed.tech/sources/datadog-the-monitor-blog.md>)

Topics: [Azure](<https://devfeed.tech/topics/azure.md>), [datadog](<https://devfeed.tech/topics/datadog.md>), [agent observability](<https://devfeed.tech/topics/agent-observability.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [Azure OpenAI](<https://devfeed.tech/topics/azure-openai.md>), [bits ai](<https://devfeed.tech/topics/bits-ai.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-stack](<https://devfeed.tech/tags/ai-stack.md>), [ai-workloads](<https://devfeed.tech/tags/ai-workloads.md>), [azure](<https://devfeed.tech/tags/azure.md>), [azure-ai](<https://devfeed.tech/tags/azure-ai.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cost-management](<https://devfeed.tech/tags/cost-management.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [datadog](<https://devfeed.tech/tags/datadog.md>), [ignite](<https://devfeed.tech/tags/ignite.md>), [integrations](<https://devfeed.tech/tags/integrations.md>), [investigation](<https://devfeed.tech/tags/investigation.md>), [issue](<https://devfeed.tech/tags/issue.md>), [management](<https://devfeed.tech/tags/management.md>), [microsoft](<https://devfeed.tech/tags/microsoft.md>), [microsoft-azure](<https://devfeed.tech/tags/microsoft-azure.md>), [microsoft-ignite](<https://devfeed.tech/tags/microsoft-ignite.md>), [microsoft-ignite-2026](<https://devfeed.tech/tags/microsoft-ignite-2026.md>), [monitor](<https://devfeed.tech/tags/monitor.md>), [stack](<https://devfeed.tech/tags/stack.md>)

### AI overview

Datadog's expanded Azure coverage combines AI workload monitoring, cost analysis, and service-level telemetry to help teams investigate performance and spending. Azure OpenAI dashboards and recommended monitors cover blocked calls, deployment capacity, failures, latency, and token use; Agent Observability traces conversations, model calls, tools, and retrieval steps, while integrations monitor dependencies such as Azure AI Search, Cosmos DB for MongoDB vCore, and Azure Redis Cache. Cloud Cost Management adds Microsoft Foundry spending analysis by provider, model, and user, investigation of cost changes, and tracking for Azure commitments. Azure advanced platform metrics expose Blob Storage usage and blob counts by container after enablement, allowing teams to rank growth and alert on container capacity; retained and soft-deleted blobs still consume capacity. The announced 55 integrations extend Azure Monitor coverage across multiple service areas. Observability Pipelines' generally available Azure Data Explorer destination routes structured logs directly to that service without an Event Hubs hop, alongside separate destinations for Sentinel and Datadog.

### Source excerpt

At Microsoft Ignite 2026, Datadog is announcing expanded Azure monitoring with 55 new integrations for AI workloads, cost management, and issue investigation.

[Next page](<https://devfeed.tech/topics/llm-observability.md?cursor=WyIyMDI2LTExLTE3VDAwOjAwOjAwKzAwOjAwIiwgIjAwNTU0Mjk5LTZkYWItNGUzMi1iMTRlLTlhYzY3ZTRkODY3NSJd>)