# reliability-engineering

A branch of systems engineering focused on ensuring products, components, and systems perform intended functions without failure under defined conditions, using statistical analysis and engineering methods across hardware and software.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## HubSpot Incident Report: September 8, 2026

DevFeed: [HubSpot Incident Report: September 8, 2026](<https://devfeed.tech/articles/hubspot-incident-report-september-8-2026-142077.md>)

Original publisher: [Read original article](<https://product.hubspot.com/blog/hubspot-incident-report-september-8-2026>)

Author: Karina Devadoss

Published: 2026-11-19T00:09:28Z

Content type: news

Language: en

Sources: [HubSpot](<https://devfeed.tech/sources/hubspot.md>)

Topics: [Post Mortem](<https://devfeed.tech/topics/post-mortem.md>), [incident management](<https://devfeed.tech/topics/incident-management.md>), [hubspot](<https://devfeed.tech/topics/hubspot.md>), [reliability-engineering](<https://devfeed.tech/topics/reliability-engineering.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [ai](<https://devfeed.tech/tags/ai.md>), [alerting](<https://devfeed.tech/tags/alerting.md>), [audit](<https://devfeed.tech/tags/audit.md>), [cache](<https://devfeed.tech/tags/cache.md>), [crm](<https://devfeed.tech/tags/crm.md>), [dashboards](<https://devfeed.tech/tags/dashboards.md>), [database](<https://devfeed.tech/tags/database.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [fallback](<https://devfeed.tech/tags/fallback.md>), [guardrail](<https://devfeed.tech/tags/guardrail.md>), [hubspot](<https://devfeed.tech/tags/hubspot.md>), [incident-report](<https://devfeed.tech/tags/incident-report.md>), [north-america](<https://devfeed.tech/tags/north-america.md>), [outage](<https://devfeed.tech/tags/outage.md>)

### AI overview

A failed permissions-cache update job triggered a 39-minute outage for HubSpot customers in North America on September 8, 2026; no data was lost. A database query guardrail caused the job to fail, and when the cache exceeded its 10-minute staleness threshold, authorization checks fell through to slower database queries. The resulting load exhausted permissions-service thread pools, causing failures across CRM, Help Desk, Inbox, Workflows, and dashboards. Engineers mitigated the impact by raising the staleness threshold, then fixed and restarted the job. HubSpot says it has changed cache updates to use pagination, audited critical services for unbounded queries, removed database fallbacks in favor of a highly available read cache, and is adding alerts for cache staleness and failed publishing jobs.

### Source excerpt

On September 8, 2026, between 11:40 AM and 12:19 PM EDT (UTC-4), HubSpot customers in the North America region experienced an outage across HubSpot tooling. No data was lost during this incident. After completing a thorough analysis of this incident, here is a detailed account of what happened, why it occurred, and the steps we are taking to prevent similar issues in the future.

## Thoughts on Verification Engineering in the Agentic SDLC

DevFeed: [Thoughts on Verification Engineering in the Agentic SDLC](<https://devfeed.tech/articles/thoughts-on-verification-engineering-in-the-agentic-sdlc-142073.md>)

Original publisher: [Read original article](<https://www.dataengineeringweekly.com/p/thoughts-on-verification-engineering>)

Author: Ananth Packkildurai

Published: 2026-11-18T21:59:55Z

Content type: article

Language: en

Sources: [Data Engineering Weekly](<https://devfeed.tech/sources/data-engineering-weekly.md>)

Topics: [Software Engineering](<https://devfeed.tech/topics/software-engineering.md>), [reliability-engineering](<https://devfeed.tech/topics/reliability-engineering.md>), [observability](<https://devfeed.tech/topics/observability.md>), [Spec Driven Development](<https://devfeed.tech/topics/spec-driven-development.md>), [performance-engineering](<https://devfeed.tech/topics/performance-engineering.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-sdlc](<https://devfeed.tech/tags/agentic-sdlc.md>), [agentic-software](<https://devfeed.tech/tags/agentic-software.md>), [agentic-software-development](<https://devfeed.tech/tags/agentic-software-development.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [ai-coding-assistants](<https://devfeed.tech/tags/ai-coding-assistants.md>), [code](<https://devfeed.tech/tags/code.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [manufacturing](<https://devfeed.tech/tags/manufacturing.md>), [sdlc](<https://devfeed.tech/tags/sdlc.md>), [software](<https://devfeed.tech/tags/software.md>), [software-supply-chain](<https://devfeed.tech/tags/software-supply-chain.md>), [supply-chain](<https://devfeed.tech/tags/supply-chain.md>), [thoughts](<https://devfeed.tech/tags/thoughts.md>), [verification](<https://devfeed.tech/tags/verification.md>)

### AI overview

The article argues that AI accelerates code production faster than existing review and testing loops can verify it, shifting the engineering bottleneck toward proving behavior before release. Static analysis and human diff reviews may miss cross-service runtime failures, while relying on canaries risks exposing users to too many regressions. It proposes Verification Engineering built around disposable production-equivalent sandboxes, synthetic test data shaped by domain distributions and failure patterns, automatically generated test harnesses and correctness/performance/resource assertions, and simulated end-to-end user workflows. Requirements and designs would become executable specifications, while production logs, traces, and interactions would feed anomalies back into future sandbox tests. These are presented as a proposed discipline, not as a comparison of specific products or evidence of measured results.

### Source excerpt

When AI accelerates code manufacturing, the software supply chain bottleneck shifts from writing code to building the feedback loop that proves it works.

## Formal Analysis of a Fault-Tolerant Extension of the ROSACE Flight Controller

DevFeed: [Formal Analysis of a Fault-Tolerant Extension of the ROSACE Flight Controller](<https://devfeed.tech/articles/formal-analysis-of-a-fault-tolerant-extension-of-the-rosace-flight-controller-139603.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22394>)

Author: Marco Bozzano (Fondazione Bruno Kessler), Alessandro Cimatti (Fondazione Bruno Kessler), Moussa Fall (Fondazione Bruno Kessler, University of Trento), Stefano Tonetta (Fondazione Bruno Kessler)

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Formal verification](<https://devfeed.tech/topics/formal-verification.md>), [Model driven development](<https://devfeed.tech/topics/model-driven-development.md>), [reliability-engineering](<https://devfeed.tech/topics/reliability-engineering.md>), [Tool](<https://devfeed.tech/topics/tool.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [analyze](<https://devfeed.tech/tags/analyze.md>), [approach](<https://devfeed.tech/tags/approach.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [autonomous](<https://devfeed.tech/tags/autonomous.md>), [based](<https://devfeed.tech/tags/based.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [builds](<https://devfeed.tech/tags/builds.md>), [cases](<https://devfeed.tech/tags/cases.md>), [component](<https://devfeed.tech/tags/component.md>), [control](<https://devfeed.tech/tags/control.md>), [controller](<https://devfeed.tech/tags/controller.md>), [controllers](<https://devfeed.tech/tags/controllers.md>), [coupling](<https://devfeed.tech/tags/coupling.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [cs-sc](<https://devfeed.tech/tags/cs-sc.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [design](<https://devfeed.tech/tags/design.md>), [extension](<https://devfeed.tech/tags/extension.md>), [flight](<https://devfeed.tech/tags/flight.md>), [formal](<https://devfeed.tech/tags/formal.md>)

### AI overview

The study extends the ROSACE flight-control benchmark with an FDIR component based on Triple Modular Redundancy to address faults such as sensor noise and sensor failures. Using COMPASTA, a model-based design tool built on ESA's TASTE, the authors model and inject faults, then use a simulation-based workflow to generate and run tests according to user-defined coverage. They construct formal models with faults and FDIR and analyze effects on system properties by coupling those models with the Simulink models used in simulation.

### Source excerpt

arXiv:2611.22394v1 Announce Type: new Abstract: Autonomous avionics controllers must satisfy stringent real-time functional and performance require- ments even in the presence of run-time faults. In this paper, we analyze ROSACE, an open-source control engineering benchmark for flight control systems, focusing on its run-time resilience against faults such as sensor noise and sensor failures. We extend the benchmark with an FDIR (Fault Detection, Identification and Recovery) component based on Triple Modular Redundancy. Our approach builds upon COMPASTA, a tool for Model-Based Design developed on top of the TASTE tool by ESA. COMPASTA enables the modeling and injection of faults into a system's design. Moreover, it implements a simulation-based verification and validation workflow, automating the generation and execution of a set of test cases, based on a user-defined notion of coverage. Using COMPASTA, we construct a set of formal models, with faults and with FDIR, and we formally analyze their impact on the system properties. The analysis is based on the coupling of the formal models with the Simulink models used for the simulations.

## Provable Mixing Properties of Parallel Mixnets Revisited

DevFeed: [Provable Mixing Properties of Parallel Mixnets Revisited](<https://devfeed.tech/articles/provable-mixing-properties-of-parallel-mixnets-revisited-141496.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22732>)

Author: Pierfrancesco Ingo, Peter Druschel, Deepak Garg

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [network security](<https://devfeed.tech/topics/network-security.md>), [tor](<https://devfeed.tech/topics/tor.md>), [routing-system](<https://devfeed.tech/topics/routing-system.md>), [reliability-engineering](<https://devfeed.tech/topics/reliability-engineering.md>)

Tags: [anonymity](<https://devfeed.tech/tags/anonymity.md>), [bandwidth](<https://devfeed.tech/tags/bandwidth.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [differential-privacy](<https://devfeed.tech/tags/differential-privacy.md>), [latency](<https://devfeed.tech/tags/latency.md>), [mixing](<https://devfeed.tech/tags/mixing.md>), [parallel](<https://devfeed.tech/tags/parallel.md>)

### AI overview

The paper analyzes how horizontally scalable parallel mixnets can provide formal message indistinguishability without adding noise or chaff messages. It bounds the variance of an adversary's posterior likelihood of a message's location, then applies concentration bounds to show that, under reasonable operating conditions, the mixnets meet a differential-indistinguishability guarantee with small ε and very small failure probability δ. The model also lets the authors examine trade-offs among anonymity, latency, and per-router bandwidth. The supplied abstract does not give specific operating thresholds or quantitative trade-off results.

### Source excerpt

arXiv:2611.22732v1 Announce Type: new Abstract: A mix-network (mixnet) is a routing network that breaks source-to-destination message trackability by indistinguishably mixing the routes of concurrently transmitted messages, thus providing anonymity to senders, receivers and sender-receiver pairs. Given their potential use in high-stakes scenarios like protecting whistleblowers, one would like to prove formally that sender anonymity holds in a mixnet, even against strong adversaries that can compromise subsets of mixnet servers. However, existing work on anonymity in mixnets is either limited to linear mixnets (which do not scale horizontally), relies on special provisions such as additional noise or chaff messages, or is empirical -- typically based on entropy, which does not provide strong guarantees if the adversary has auxiliary information. In this paper, we present a new analysis of parallel mixnets which scale horizontally, showing that these networks can provide strong indistinguishability for messages even without additional noise messages. Our analysis works by bounding the statistical variance of the adversary's posterior likelihood of where a message of interest is at a given point in time. Using concentration bounds on this distribution, we show that parallel mixnets satisfy $(\varepsilon, \delta)$-differential indistinguishability for messages routed through the mixnet (similar to $(\varepsilon, \delta)$-differential privacy) with very small failure probabilities $\delta$ and small privacy parameters $\varepsilon$ under reasonable operating conditions. Our model also enables an examination of the quantitative trade-off between anonymity, latency and per-router bandwidth in parallel mixnets.

## VoxelSBFL: Voxel-Level Spectrum-Based Failure Prediction for Vision-Language-Action Models

DevFeed: [VoxelSBFL: Voxel-Level Spectrum-Based Failure Prediction for Vision-Language-Action Models](<https://devfeed.tech/articles/voxelsbfl-voxel-level-spectrum-based-failure-prediction-for-vision-language-action-models-139607.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22453>)

Author: Arusa Kanwal, Pablo Valle, Paola Roggero, Miren Illarramendi, Shaukat Ali, Aitor Arrieta

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Amazon robotics research](<https://devfeed.tech/topics/amazon-robotics-research.md>), [Predictive AI](<https://devfeed.tech/topics/predictive-ai.md>), [reliability-engineering](<https://devfeed.tech/topics/reliability-engineering.md>), [vlm](<https://devfeed.tech/topics/vlm.md>)

Tags: [assessment](<https://devfeed.tech/tags/assessment.md>), [black-box](<https://devfeed.tech/tags/black-box.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [cs-ro](<https://devfeed.tech/tags/cs-ro.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [failure](<https://devfeed.tech/tags/failure.md>), [language](<https://devfeed.tech/tags/language.md>), [localization](<https://devfeed.tech/tags/localization.md>), [maps](<https://devfeed.tech/tags/maps.md>), [models](<https://devfeed.tech/tags/models.md>), [performance](<https://devfeed.tech/tags/performance.md>), [policy](<https://devfeed.tech/tags/policy.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [roc](<https://devfeed.tech/tags/roc.md>), [roc-auc](<https://devfeed.tech/tags/roc-auc.md>), [score](<https://devfeed.tech/tags/score.md>), [simulation](<https://devfeed.tech/tags/simulation.md>), [spectrum](<https://devfeed.tech/tags/spectrum.md>), [vision](<https://devfeed.tech/tags/vision.md>), [vision-language-action](<https://devfeed.tech/tags/vision-language-action.md>), [vision-language-action-models](<https://devfeed.tech/tags/vision-language-action-models.md>), [voxel](<https://devfeed.tech/tags/voxel.md>)

### AI overview

VoxelSBFL predicts failure risk in unseen VLA-model executions by mapping recorded tool-center-point trajectories to binary workspace-voxel coverage, then combining coverage with execution outcomes in established spectrum-based suspiciousness formulas. This black-box method needs no internal model access. In simulation, evaluation across four VLA models and 96,000 Pick-Up trajectories found an overall macro ROC-AUC of 0.704; the leading techniques significantly outperformed eight action- and trajectory-based baselines. Performance varied with voxel spatial granularity and training-set size. The results support spatial spectra for execution assessment, while runtime monitoring and policy improvement are proposed as future research.

### Source excerpt

arXiv:2611.22453v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models integrate visual perception, language understanding, and action generation to control robots. Testing these systems produces trajectories and task outcomes, but existing approaches provide limited guidance on using these recorded trajectories to assess subsequent executions or inform policy repair. We conjecture that, for a given policy and task, traversing certain workspace regions is associated with a higher likelihood of failure. To investigate this conjecture, we propose VoxelSBFL, a black-box adaptation of Spectrum-Based Fault Localization (SBFL) in which workspace voxels replace program elements as units of analysis. Recorded tool-center-point (TCP) trajectories provide binary voxel coverage, which is combined with execution outcomes to construct suspiciousness maps using established SBFL formulas. These maps identify failure-associated regions and score unseen trajectories without requiring internal model access. We evaluate 34 spectrum-based techniques across four VLA models using 96,000 Pick-Up trajectories in simulation and compare the leading techniques against eight action- and trajectory-based metrics. VoxelSBFL obtains an overall macro ROC-AUC of 0.704, significantly outperforming state-of-the art baselines. Sensitivity analyses show that performance varies with spatial granularity and training-set size. These findings support spatial spectra for VLA execution assessment and motivate future research on runtime monitoring, self-healing and targeted policy improvement.

## Operating-Point Failure of Frozen Guardrail Probes Under Backbone Updates

DevFeed: [Operating-Point Failure of Frozen Guardrail Probes Under Backbone Updates](<https://devfeed.tech/articles/operating-point-failure-of-frozen-guardrail-probes-under-backbone-updates-139270.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.19216>)

Author: Yusuf Khalid Shire, Sang-Chul Kim

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [LLM security](<https://devfeed.tech/topics/llm-security.md>), [reliability-engineering](<https://devfeed.tech/topics/reliability-engineering.md>)

Tags: [attack](<https://devfeed.tech/tags/attack.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [failure](<https://devfeed.tech/tags/failure.md>), [guardrail](<https://devfeed.tech/tags/guardrail.md>), [language](<https://devfeed.tech/tags/language.md>), [model](<https://devfeed.tech/tags/model.md>)

### AI overview

Across 18 cells covering 16 public language-model updates, frozen guardrail probes often lost their intended 1% false-positive operating point even when AUROC barely changed (at most 0.0058). In the first phase, 8 of 9 cells failed closed; in the second, 3 of 9 failed either closed or open, though the phases differed in update type and input format. Benchmark staleness criteria missed all affected cells, while a label-free rate estimate flagged 91.8%; a two-moment model matched rate changes within a factor of 1.5 in 90.5% of 63 registered cells. Resetting the threshold with stored benign prompts restored the false-positive rate without new labels, but poisoning 1% of prompts left at most 1.7% of harmful prompts flagged. An attacker controlling the update cut detection of a targeted category in three of five seeds; both checks missed the attack, while a post hoc labelled check detected it.

### Source excerpt

arXiv:2611.19216v1 Announce Type: new Abstract: We freeze a guardrail probe and threshold, then measure its operating point across 18 cells covering 16 public language-model updates. At a 1% target, the threshold fails closed in 8 of 9 cells in the first phase and either fails closed or open in 3 of 9 cells in a registered second phase (descriptive). The two phases also differ in update type and input format. The largest increase is from 1.01% to 15.12%, while the area under the receiver operating characteristic curve changes by no more than 0.0058 in any of these cells. A benchmark's staleness criteria classify none of them as stale. A rule based on these criteria flags 0.019 of layer cells where the rate doubles or halves. A label-free rate estimate flags 0.918, clearing 0.898 of the remaining cells. A two-moment model matches the change within a factor of 1.5 in 0.905 of 63 registered cells and in 0.773 of the 22 cells that moved. Resetting the threshold using stored benign prompts restores the false-positive rate without requiring new labels. However, poisoning 1% of these prompts leaves at most 0.017 of harmful prompts flagged. An attacker controlling the update cut detection of one of three targeted categories in three of five seeds. Both checks missed this attack, while a post hoc labelled check detected it.

## SREGym: A Live Benchmark for AI SRE Agents with High-Fidelity Failure Scenarios

DevFeed: [SREGym: A Live Benchmark for AI SRE Agents with High-Fidelity Failure Scenarios](<https://devfeed.tech/articles/sregym-a-live-benchmark-for-ai-sre-agents-with-high-fidelity-failure-scenarios-137251.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.07161>)

Author: Jackson Clark, Yiming Su, Saad Mohammad Rafid Pial, Yifang Tian, Lily Gniedziejko, Hans-Arno Jacobsen, Yinfang Chen, Tianyin Xu

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Production Engineering](<https://devfeed.tech/topics/production-engineering.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [reliability-engineering](<https://devfeed.tech/topics/reliability-engineering.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [failure](<https://devfeed.tech/tags/failure.md>), [live](<https://devfeed.tech/tags/live.md>), [sre](<https://devfeed.tech/tags/sre.md>)

### AI overview

SREGym provides a live benchmark for evaluating AI agents that diagnose and mitigate production system failures. Built on real-world cloud-native stacks, it uses modular fault and noise injectors to simulate faults across layers, ambient noise, and failure modes including metastable and correlated failures. The benchmark currently contains 90 SRE problems. Evaluation of frontier agents found performance varied by failure type, with up to 40% differences in end-to-end results. The supplied abstract does not specify the agents, evaluation setup, or how the reported differences were measured.

### Source excerpt

arXiv:2605.07161v4 Announce Type: replace Abstract: AI agents are increasingly used to diagnose and mitigate failures in production systems, known as agentic Site Reliability Engineering (SRE). Current SRE benchmarks are limited to oversimplistic SRE tasks and are unfortunately hard to extend due to bespoke designs. We present SREGym, a high-fidelity benchmark for SRE agents. SREGym exposes a live system environment built atop real-world cloud-native system stacks, where high-fidelity failure scenarios are simulated through fault injectors. SREGym models the complexity of production environments by simulating (1) a wide range of faults at different layers, (2) various ambient noises, and (3) diverse failure modes such as metastable failures and correlated failures. SREGym is architected as a modular, extensible framework that orchestrates fault and noise injectors across stacks. SREGym currently includes 90 realistic, challenging SRE problems. We use SREGym to evaluate frontier agents and show that their capabilities varies significantly in addressing different kinds of failures, with up to 40% differences in end-to-end results. SREGym is actively maintained as an open-source project and has been used by researchers and practitioners.

## Three Invisible Failure Modes in Speech-Based Age Estimation for Child Safety

DevFeed: [Three Invisible Failure Modes in Speech-Based Age Estimation for Child Safety](<https://devfeed.tech/articles/three-invisible-failure-modes-in-speech-based-age-estimation-for-child-safety-137370.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18971>)

Author: Christopher M. Stewart, Kelsey Kraus, J. Elliott Casal, Geoffrey L. Brown

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [LLM security](<https://devfeed.tech/topics/llm-security.md>), [reliability-engineering](<https://devfeed.tech/topics/reliability-engineering.md>), [ai observability](<https://devfeed.tech/topics/ai-observability.md>)

Tags: [age-assurance](<https://devfeed.tech/tags/age-assurance.md>), [ai](<https://devfeed.tech/tags/ai.md>), [child-safety](<https://devfeed.tech/tags/child-safety.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-cy](<https://devfeed.tech/tags/cs-cy.md>), [cs-sd](<https://devfeed.tech/tags/cs-sd.md>), [failure](<https://devfeed.tech/tags/failure.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [speech](<https://devfeed.tech/tags/speech.md>)

### AI overview

Across five studies using two speech corpora of children under 13, three audio-capable language models classified children as over 13 in 2.5%-42.5% of responses; every model that returned an age placed some 5- to 11-year-olds over 18. The study identifies pipeline failures that standard monitoring would miss: a client library silently double-encoded audio, prompting implausible age estimates and hallucinated explanations; a replaced endpoint refused nearly all age questions; and models sometimes gave inconsistent age classifications, including threshold answers that contradicted numeric estimates. None triggered warnings, status codes, or output-format changes. The authors argue that child-safety age-assurance claims need pipeline-level evaluation and reporting practices that expose these failures.

### Source excerpt

arXiv:2611.18971v1 Announce Type: new Abstract: Child safety in AI depends on the accurate identification of minor users. Young children tend to talk rather than type when interacting with AI, so in practice this identification requires predicting age from speech. We evaluated whether three audio-capable large language models can predict a child's age from their speech across five studies using two speech corpora of children under the age of 13. Models classified children as over 13 on 2.5% to 42.5% of audio responses, and every model that answered placed some 5- to 11-year-olds over 18. While measuring these rates, we uncovered three distinct failure modes. A client library silently double-encoded speech samples, leading one model to classify children aged 5 to 10 years old as being in their thirties and forties with hallucinated explanations. A second model's endpoint was replaced by a successor that refused nearly every age-related question and never produced an age. Every model sometimes placed the same speech sample under 13 and over 18 years of age, but a third model went further, giving threshold answers that contradicted its own numeric estimates. None of these failure modes produced a warning, a status code, or a change in output format, so they would all pass standard monitoring. These results demonstrate why child-safety claims about age assurance should be evaluated at the level of the pipeline rather than the model. We describe the reporting practices that would make such failures detectable.

## Selection-Aware Certification of Client-Constrained Fraud Rule Portfolios Under Synthetic Distribution Shift

DevFeed: [Selection-Aware Certification of Client-Constrained Fraud Rule Portfolios Under Synthetic Distribution Shift](<https://devfeed.tech/articles/selection-aware-certification-of-client-constrained-fraud-rule-portfolios-under-synthetic-distribution-shift-137324.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18687>)

Author: Rishabh Pande, Yu Zeng

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [reliability-engineering](<https://devfeed.tech/topics/reliability-engineering.md>), [fake account fraud](<https://devfeed.tech/topics/fake-account-fraud.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [account](<https://devfeed.tech/tags/account.md>), [adjustment](<https://devfeed.tech/tags/adjustment.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [authors](<https://devfeed.tech/tags/authors.md>), [bank](<https://devfeed.tech/tags/bank.md>), [base](<https://devfeed.tech/tags/base.md>), [bounded](<https://devfeed.tech/tags/bounded.md>), [candidate](<https://devfeed.tech/tags/candidate.md>), [case](<https://devfeed.tech/tags/case.md>), [case-study](<https://devfeed.tech/tags/case-study.md>), [certification](<https://devfeed.tech/tags/certification.md>), [changes](<https://devfeed.tech/tags/changes.md>), [client](<https://devfeed.tech/tags/client.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [fraud](<https://devfeed.tech/tags/fraud.md>), [portfolios](<https://devfeed.tech/tags/portfolios.md>), [q-fin-rm](<https://devfeed.tech/tags/q-fin-rm.md>), [selection](<https://devfeed.tech/tags/selection.md>), [shift](<https://devfeed.tech/tags/shift.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>)

### AI overview

For bounded OR fraud-rule portfolios, one-sided Clopper-Pearson lower bounds with Bonferroni adjustment reduced held-out precision or recall floor violations in the evaluated synthetic settings, at the cost of lower deployment. Across 6,400 datasets from two author-designed generators, full-validation certification cut violations by an average of 29.1 percentage points versus point-estimate selection and 22.2 points versus a fixed 0.05 margin, while reducing deployment by 31.7 and 29.4 points, respectively; results varied by condition. Using all rather than half the validation data raised deployment by 17.1 points on average, without uniformly reducing violations under shift. In a separate no-shift sweep, deployment increased with validation size, from 61.3% at 1,500 rows to 90.2% at 15,000; no violations occurred among 2,887 deployed portfolios on 100,000-row holdouts. A temporal study on a public synthetic Bank Account Fraud Base dataset applied recall subject to a 5% false-positive-rate cap, but its candidate pool captured under 1% of test fraud. These findings cover the tested generators and synthetic case study, not real transaction populations or arbitrary distribution shifts

### Source excerpt

arXiv:2611.18687v1 Announce Type: cross Abstract: Selecting fraud-rule portfolios to satisfy a precision or recall floor can overfit the data used for selection. We study finite-family certification for bounded OR portfolios, comparing point-estimate selection, a fixed safety margin, and one-sided Clopper-Pearson lower bounds with Bonferroni adjustment. In a factorial follow-up with 6,400 datasets from two author-designed synthetic generators, full-validation certification reduced the held-out floor-violation rate by 29.1 percentage points on average versus full-validation point selection, while reducing deployment by 31.7 points. Compared with a fixed 0.05 margin, certification reduced violations by 22.2 points and deployment by 29.4 points; results varied across conditions. Using the full rather than half validation sample increased deployment by 17.1 points on average, without a uniform reduction in shifted-test violations. In a separate 400-dataset no-shift sweep, certification deployment rose from 61.3% to 90.2% as validation size increased from 1,500 to 15,000; no violations occurred among 2,887 deployed portfolios tested on 100,000-row holdouts. A temporal case study on the public synthetic Bank Account Fraud Base dataset used recall at a 5% false-positive-rate cap; the selected candidate pool captured less than 1% of test fraud, showing limited coverage in this setup. Together, these experiments characterize how finite-family certification changes the reliability-deployment tradeoff for bounded OR portfolios under specified synthetic conditions. The evidence covers the evaluated generators and one public synthetic BAF case study; it does not estimate performance on real transaction populations or robustness to arbitrary distribution shift. The authors revised and rewrote this abstract after receiving a draft from OpenAI Codex.

## A Hierarchical Statistical Framework for Evaluating Agent Reliability

DevFeed: [A Hierarchical Statistical Framework for Evaluating Agent Reliability](<https://devfeed.tech/articles/a-hierarchical-statistical-framework-for-evaluating-agent-reliability-137472.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.19614>)

Author: Mirco Milletar\\\`i, Tuhin Kundu

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [reliability-engineering](<https://devfeed.tech/topics/reliability-engineering.md>), [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>)

Tags: [agent-reliability](<https://devfeed.tech/tags/agent-reliability.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-systems](<https://devfeed.tech/tags/agentic-systems.md>), [aggregation](<https://devfeed.tech/tags/aggregation.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cond-mat-stat-mech](<https://devfeed.tech/tags/cond-mat-stat-mech.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [framework](<https://devfeed.tech/tags/framework.md>), [hierarchy](<https://devfeed.tech/tags/hierarchy.md>), [llm](<https://devfeed.tech/tags/llm.md>), [math-pr](<https://devfeed.tech/tags/math-pr.md>), [metric](<https://devfeed.tech/tags/metric.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [model-performance](<https://devfeed.tech/tags/model-performance.md>), [noise](<https://devfeed.tech/tags/noise.md>), [pass](<https://devfeed.tech/tags/pass.md>), [performance](<https://devfeed.tech/tags/performance.md>), [physics](<https://devfeed.tech/tags/physics.md>), [policy](<https://devfeed.tech/tags/policy.md>), [production](<https://devfeed.tech/tags/production.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [sample](<https://devfeed.tech/tags/sample.md>)

### AI overview

The paper proposes a hierarchical framework for measuring agent reliability across scenarios, instances, and repeated samples. It applies uncertainty estimates to mean pass rate, pass@k, and passhat^k, showing how aggregation at different levels exposes different properties of agent performance. The framework connects this aggregation to coarse-graining and separates execution noise from task heterogeneity. It also derives exact finite-sample sampling variances for unbiased pass@k and passhat^k estimators. The supplied text is an abstract, so it does not provide implementation details or empirical results beyond the stated motivation.

### Source excerpt

arXiv:2611.19614v1 Announce Type: new Abstract: Agents operating in complex scenarios often face uncertainty due to incomplete information about the environment they operate in, ambiguous user queries, complex business rules and the inherently stochastic dynamics of the governing policy. Several new benchmarks have been proposed to evaluate LLM-based agents on complex tasks in multi-tool environments, including settings with policy constraints and varying levels of intent ambiguity. These conditions make it important to evaluate agentic systems systematically and quantify their performance under different conditions to assess their reliability in production. This need is evident in the recently introduced Thinkingbox benchmark, where even the strongest models struggle to perform consistently across diverse business scenarios. This paper presents a statistical framework for evaluating uncertainty and reliability in agentic systems. We define a hierarchy of scenarios, instances, and samples to assess model performance and variability systematically. We apply this framework to common metrics found in the literature: mean pass rate, $pass@k$, and $passhat^k$, with uncertainty estimates for each metric. Aggregating metrics at different levels of the hierarchy reveals different structures and properties of the policy model; we connect this hierarchical aggregation with the known process of \emph{coarse-graining} as intended in the physics and statistical literature. We independently derive exact finite-sample expressions for the sampling variances of the unbiased $pass@k$ and $passhatk$ estimators; related results also appear in recent submissions. Our central contribution is a constructive derivation of hierarchical uncertainty propagation, making explicit its correspondence with coarse-graining and separating execution noise from task heterogeneity.

## eIRWR: Enhanced Iterative Random Walk with Restart for Scalable Root Cause Analysis in Microservices

DevFeed: [eIRWR: Enhanced Iterative Random Walk with Restart for Scalable Root Cause Analysis in Microservices](<https://devfeed.tech/articles/eirwr-enhanced-iterative-random-walk-with-restart-for-scalable-root-cause-analysis-in-microservices-136966.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2608.08073>)

Author: Saiful Khan, Afrah Farea

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Graph Mining](<https://devfeed.tech/topics/graph-mining.md>), [reliability-engineering](<https://devfeed.tech/topics/reliability-engineering.md>)

Tags: [alibaba](<https://devfeed.tech/tags/alibaba.md>), [cascade](<https://devfeed.tech/tags/cascade.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [cs-sy](<https://devfeed.tech/tags/cs-sy.md>), [dependency-graph](<https://devfeed.tech/tags/dependency-graph.md>), [eess-sy](<https://devfeed.tech/tags/eess-sy.md>), [loops](<https://devfeed.tech/tags/loops.md>), [microservices](<https://devfeed.tech/tags/microservices.md>), [mrr](<https://devfeed.tech/tags/mrr.md>), [online](<https://devfeed.tech/tags/online.md>), [random-walk](<https://devfeed.tech/tags/random-walk.md>), [root-cause-analysis](<https://devfeed.tech/tags/root-cause-analysis.md>)

### AI overview

eIRWR addresses a weakness in anomaly-based random walks: raw restart scores can favor highly anomalous downstream victims over the less anomalous root cause. The authors find that basic IRWR's resilience damping is equivalent to increasing the restart probability in Personalized PageRank (PPR), and use restart-tuned PPR as a fairer baseline. eIRWR additionally sharpens restart mass toward suspicious nodes and adds self-loops and backward edges to help probability accumulate at cascade sources. On three 12K-25K-node topologies from the Alibaba Microservice Trace Dataset, it reports the highest accuracy among compared graph-based methods at each tested root-cause visibility level. At visibility 0.3, MRR is 0.78; at high visibility, MRR is 0.95, with runtime under 30 ms on 17K-node graphs. These results are from the stated dataset and comparisons.

### Source excerpt

arXiv:2608.08073v2 Announce Type: replace Abstract: Root cause analysis (RCA) in microservice systems must find the faulty service behind symptoms that spread across hundreds or thousands of dependent services. Graph-based random walk methods propagate anomaly evidence over the service dependency graph. However, existing anomaly-restart walks restart from the raw anomaly vector, which is often dominated by strongly anomalous downstream victims rather than the less anomalous source. Through a controlled ablation of a basic anomaly-restart walk, the Iterative Random Walk with Restart (IRWR), we show that its ``resilience damping'' is equivalent to a higher restart probability in Personalized PageRank (PPR). Basic IRWR is therefore no stronger than a restart-tuned PPR, which we adopt as a fairer baseline. We then present Enhanced IRWR (eIRWR), which goes beyond restart tuning. It (a)~concentrates restart mass on the most suspicious nodes through power-law sharpening and (b)~adds self-loops and backward edges so that probability accumulates at cascade sources. On three large topologies (12K--25K nodes) from the Alibaba Microservice Trace Dataset, eIRWR is the most accurate of the compared graph-based methods at every level of root-cause visibility. At visibility $v = 0.3$, it reaches a Mean Reciprocal Rank (MRR) of $0.78$, about four times a restart-tuned PPR and nine times the best other baseline. At high visibility, it reaches MRR\,$= 0.95$ and runs in under 30\,ms on graphs with 17K nodes, so it is suitable for online use.

## CLEW: Log-Anchored Backward Tracing for Root-Cause Code Localization

DevFeed: [CLEW: Log-Anchored Backward Tracing for Root-Cause Code Localization](<https://devfeed.tech/articles/clew-log-anchored-backward-tracing-for-root-cause-code-localization-137449.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.19448>)

Author: Wenji Bai, Muhammad Waseem, Jaakko Peltonen, Pekka Abrahamsson

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [atlassian automated rca](<https://devfeed.tech/topics/atlassian-automated-rca.md>), [reliability-engineering](<https://devfeed.tech/topics/reliability-engineering.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [approach](<https://devfeed.tech/tags/approach.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [candidate](<https://devfeed.tech/tags/candidate.md>), [cause](<https://devfeed.tech/tags/cause.md>), [code](<https://devfeed.tech/tags/code.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [defect](<https://devfeed.tech/tags/defect.md>), [diagnostic](<https://devfeed.tech/tags/diagnostic.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [events](<https://devfeed.tech/tags/events.md>), [exploration](<https://devfeed.tech/tags/exploration.md>), [failure](<https://devfeed.tech/tags/failure.md>), [fault-injection](<https://devfeed.tech/tags/fault-injection.md>), [function](<https://devfeed.tech/tags/function.md>), [functions](<https://devfeed.tech/tags/functions.md>), [localization](<https://devfeed.tech/tags/localization.md>), [log](<https://devfeed.tech/tags/log.md>), [root](<https://devfeed.tech/tags/root.md>), [tracing](<https://devfeed.tech/tags/tracing.md>)

### AI overview

CLEW anchors runtime log observations to symptom functions, then uses LLM-guided backward tracing to rank candidate root-cause functions and provide a proposed mechanism path linked to log events. On the 192-case LogLoc benchmark, it achieved the highest reported Acc@1 (77.3%) and MRR (81.3%) on LogLoc-Issues, and matched leading baselines at 97.6% Acc@1 on LogLoc-Airflow. Across the cases, its wall-clock time was about 73% lower than LocAgent's at similar average inference cost. The abstract reports no aggregate degradation under the tested perturbations, with modest metric changes across evaluated seeds and base models; these findings are limited to the reported evaluation conditions.

### Source excerpt

arXiv:2611.19448v1 Announce Type: cross Abstract: Failures in industrial software systems may involve multiple components and be difficult to reproduce outside the production environment. Root-cause code localization therefore often relies on issue descriptions and runtime logs. However, log evidence may be noisy or incomplete, and the code reporting a symptom may differ from the code containing the underlying defect. These conditions make it challenging to identify relevant log evidence and determine which functions could have caused the reported failure. To address this gap, we introduce CLEW, an agentic approach that combines rule-based log-to-code anchoring with LLM-guided backward code tracing. CLEW organizes selected observations into diagnostic states, guides source exploration from anchored symptom functions, and returns ranked candidate functions, an affected module, and an explanation comprising a proposed mechanism path and associated log events. For evaluation, we construct LogLoc, a benchmark comprising 192 real-world issue and fault-injection cases. Against three baselines, CLEW records the highest observed Acc@1 (77.3%) and MRR (81.3%) on LogLoc-Issues and matches the leading baselines at 97.6% Acc@1 on LogLoc-Airflow. Averaged over all 192 cases, CLEW's wall-clock time is approximately 73% lower than LocAgent's at a similar average inference cost. No aggregate degradation is observed under the tested perturbations, with modest changes in Acc@1 and MRR across the evaluated seeds and base models. These results suggest that CLEW can combine function-level localization with evidence links and runtime efficiency under the evaluated conditions.

## Microsoft outage: A cascading failure that started with an undetected bug

DevFeed: [Microsoft outage: A cascading failure that started with an undetected bug](<https://devfeed.tech/articles/microsoft-outage-a-cascading-failure-that-started-with-an-undetected-bug-136600.md>)

Original publisher: [Read original article](<https://www.networkworld.com/article/4249392/microsoft-outage-a-cascading-failure-that-started-with-an-undetected-bug-2.html>)

Author: Evan Schuman

Published: 2026-11-17T00:02:57Z

Content type: news

Language: en

Sources: [Network World](<https://devfeed.tech/sources/network-world.md>)

Topics: [Post Mortem](<https://devfeed.tech/topics/post-mortem.md>), [reliability-engineering](<https://devfeed.tech/topics/reliability-engineering.md>), [microsoft 365](<https://devfeed.tech/topics/microsoft-365.md>), [test](<https://devfeed.tech/topics/test.md>), [Load Balancing](<https://devfeed.tech/topics/load-balancing.md>), [Microsoft](<https://devfeed.tech/topics/microsoft.md>), [Microsoft Teams](<https://devfeed.tech/topics/microsoft-teams.md>)

Tags: [availability](<https://devfeed.tech/tags/availability.md>), [bug](<https://devfeed.tech/tags/bug.md>), [defect](<https://devfeed.tech/tags/defect.md>), [edition-us](<https://devfeed.tech/tags/edition-us.md>), [exchange](<https://devfeed.tech/tags/exchange.md>), [exchange-online](<https://devfeed.tech/tags/exchange-online.md>), [failure](<https://devfeed.tech/tags/failure.md>), [gateway](<https://devfeed.tech/tags/gateway.md>), [global](<https://devfeed.tech/tags/global.md>), [global-outage](<https://devfeed.tech/tags/global-outage.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [high-availability](<https://devfeed.tech/tags/high-availability.md>), [incident](<https://devfeed.tech/tags/incident.md>), [incident-report](<https://devfeed.tech/tags/incident-report.md>), [independent](<https://devfeed.tech/tags/independent.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [isolation](<https://devfeed.tech/tags/isolation.md>), [issue](<https://devfeed.tech/tags/issue.md>), [language-en](<https://devfeed.tech/tags/language-en.md>), [microsoft](<https://devfeed.tech/tags/microsoft.md>), [microsoft-outage](<https://devfeed.tech/tags/microsoft-outage.md>), [networking](<https://devfeed.tech/tags/networking.md>), [outage](<https://devfeed.tech/tags/outage.md>), [storytype-news](<https://devfeed.tech/tags/storytype-news.md>)

### AI overview

A Microsoft 365 gateway software defect, deployed globally for months without impact, surfaced when Teams failed over to the primary gateway, increasing and changing traffic while nearly 14% of gateway capacity was already unavailable. Microsoft's load tests missed the defect because they used different traffic and SKU mixes and tended to use weaker, lower-core hardware; the report says more cores and newer hardware worsened contention. Requests accumulated, and automatic rerouting spread load across fewer healthy servers, causing a fleet-wide cascade that left more than 80% unresponsive at the low point. Recovery automation and normal rollback, flighting and feature flags could not quickly restore service. The preliminary report did not explain the unrelated capacity issue, and the article notes that Microsoft had yet to provide customers concrete reassurance against recurrence.

### Source excerpt

Following a roughly day-long global outage of many Microsoft services last week, Microsoft on Monday issued a preliminary post-incident report that identified multiple causes for the issues, including the admission that the environment used to test the underlying problematic software update did not match the actual infrastructure it would run on. The report, a summary of which was posted on social media by Microsoft EVP Charles Lamanna, identified some of the factors that cascaded to cause the outage. "A latent defect caused servers to become overwhelmed and unresponsive under certain load conditions," the report said, adding, "this defect was the result of a fix that was prepared specifically for Microsoft 365 networking gateway servers." Its deployment began in July, and, the report said, "for more than three months, the change was deployed worldwide without impact." Microsoft admitted that, although it did test the change before deployment, its test environment didn't match reality. "Our normal load testing did not detect the latent issue because the different traffic and SKU mix used in our test labs," it said. "The contention issue is worsened on machines with more cores and newer hardware; whereas our load testing tended to focus on our weaker hardware SKUs with fewer cores." A surge in traffic caused by unrelated issues also helped trigger the problem and delay it from being fixed, the company explained. "Within Microsoft 365, we maintain two independent and heterogenous network gateway systems designed for isolation and high availability," Microsoft said. "This allows many Microsoft 365 services to fail to/from them during outages or disruptions. Approximately fifteen hours before the outage, Teams failed over from the alternative networking gateway to the primary gateway that also supports Exchange Online. This increased the load on the networking gateway and changed its traffic patterns." To make things worse, the report said, "nearly 14% of server capacit

## Microsoft outage: A cascading failure that started with an undetected bug

DevFeed: [Microsoft outage: A cascading failure that started with an undetected bug](<https://devfeed.tech/articles/microsoft-outage-a-cascading-failure-that-started-with-an-undetected-bug-136867.md>)

Original publisher: [Read original article](<https://www.computerworld.com/article/4249364/microsoft-outage-a-cascading-failure-that-started-with-an-undetected-bug.html>)

Author: Evan Schuman

Published: 2026-11-16T23:54:42Z

Content type: news

Language: en

Sources: [Computerworld](<https://devfeed.tech/sources/computerworld.md>)

Topics: [cloud service outage management](<https://devfeed.tech/topics/cloud-service-outage-management.md>), [reliability-engineering](<https://devfeed.tech/topics/reliability-engineering.md>), [Post Mortem](<https://devfeed.tech/topics/post-mortem.md>), [microsoft 365](<https://devfeed.tech/topics/microsoft-365.md>), [Load Balancing](<https://devfeed.tech/topics/load-balancing.md>), [test](<https://devfeed.tech/topics/test.md>), [Microsoft Teams](<https://devfeed.tech/topics/microsoft-teams.md>)

Tags: [bug](<https://devfeed.tech/tags/bug.md>), [defect](<https://devfeed.tech/tags/defect.md>), [edition-us](<https://devfeed.tech/tags/edition-us.md>), [exchange](<https://devfeed.tech/tags/exchange.md>), [exchange-online](<https://devfeed.tech/tags/exchange-online.md>), [failure](<https://devfeed.tech/tags/failure.md>), [gateway](<https://devfeed.tech/tags/gateway.md>), [global-outage](<https://devfeed.tech/tags/global-outage.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [high-availability](<https://devfeed.tech/tags/high-availability.md>), [incident](<https://devfeed.tech/tags/incident.md>), [incident-report](<https://devfeed.tech/tags/incident-report.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [isolation](<https://devfeed.tech/tags/isolation.md>), [language-en](<https://devfeed.tech/tags/language-en.md>), [load](<https://devfeed.tech/tags/load.md>), [load-testing](<https://devfeed.tech/tags/load-testing.md>), [microsoft](<https://devfeed.tech/tags/microsoft.md>), [microsoft-365](<https://devfeed.tech/tags/microsoft-365.md>), [microsoft-365-office-suites-productivity-software-vendors-and-providers](<https://devfeed.tech/tags/microsoft-365-office-suites-productivity-software-vendors-and-providers.md>), [microsoft-outage](<https://devfeed.tech/tags/microsoft-outage.md>), [network](<https://devfeed.tech/tags/network.md>), [office-suites](<https://devfeed.tech/tags/office-suites.md>), [outage](<https://devfeed.tech/tags/outage.md>), [productivity-software](<https://devfeed.tech/tags/productivity-software.md>), [storytype-news](<https://devfeed.tech/tags/storytype-news.md>), [vendors-and-providers](<https://devfeed.tech/tags/vendors-and-providers.md>)

### AI overview

Microsoft's preliminary report attributes the outage to a latent bug in a networking gateway update, exposed when Teams traffic shifted to the primary gateway and unrelated capacity loss left it under greater load. Testing missed the defect because lab traffic and hardware differed from production; the problem was worse on newer, higher-core servers. As overloaded servers stopped responding, traffic shifted onto fewer healthy servers, spreading the failure. More than 80% of servers became unresponsive at the low point, and restarting them could trigger the bug again. Automated recovery, rollback, flighting and feature flags could not quickly restore service. Consultants cited the production-testing mismatch and delayed alerting as concerns; Microsoft did not explain the unrelated capacity issue in the excerpt.

### Source excerpt

Following a roughly day-long global outage of many Microsoft services last week, Microsoft on Monday issued a preliminary post-incident report that identified multiple causes for the issues, including the admission that the environment used to test the underlying problematic software update did not match the actual infrastructure it would run on. The report, a summary of which was posted on social media by Microsoft EVP Charles Lamanna, identified some of the factors that cascaded to cause the outage. "A latent defect caused servers to become overwhelmed and unresponsive under certain load conditions," the report said, adding, "this defect was the result of a fix that was prepared specifically for Microsoft 365 networking gateway servers." Its deployment began in July, and, the report said, "for more than three months, the change was deployed worldwide without impact." Microsoft admitted that, although it did test the change before deployment, its test environment didn't match reality. "Our normal load testing did not detect the latent issue because the different traffic and SKU mix used in our test labs," it said. "The contention issue is worsened on machines with more cores and newer hardware; whereas our load testing tended to focus on our weaker hardware SKUs with fewer cores." A surge in traffic caused by unrelated issues also helped trigger the problem and delay it from being fixed, the company explained. "Within Microsoft 365, we maintain two independent and heterogenous network gateway systems designed for isolation and high availability," Microsoft said. "This allows many Microsoft 365 services to fail to/from them during outages or disruptions. Approximately fifteen hours before the outage, Teams failed over from the alternative networking gateway to the primary gateway that also supports Exchange Online. This increased the load on the networking gateway and changed its traffic patterns." To make things worse, the report said, "nearly 14% of server capacit

## Silicon Motherboard Enables Double-Sided Integration of ICs and Passive Components

DevFeed: [Silicon Motherboard Enables Double-Sided Integration of ICs and Passive Components](<https://devfeed.tech/articles/silicon-motherboard-enables-double-sided-integration-of-ics-and-passive-components-135947.md>)

Original publisher: [Read original article](<https://www.eetimes.com/silicon-motherboard-enables-double-sided-integration-of-ics-and-passive-components/>)

Author: Nisshinbo Micro Devices

Published: 2026-11-16T17:00:00Z

Content type: article

Language: en

Sources: [EE Times](<https://devfeed.tech/sources/ee-times.md>)

Topics: [reliability-engineering](<https://devfeed.tech/topics/reliability-engineering.md>)

Tags: [3d](<https://devfeed.tech/tags/3d.md>), [absolute](<https://devfeed.tech/tags/absolute.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-servers](<https://devfeed.tech/tags/ai-servers.md>), [all](<https://devfeed.tech/tags/all.md>), [allows](<https://devfeed.tech/tags/allows.md>), [also](<https://devfeed.tech/tags/also.md>), [amplifier](<https://devfeed.tech/tags/amplifier.md>), [analog](<https://devfeed.tech/tags/analog.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [area](<https://devfeed.tech/tags/area.md>), [business](<https://devfeed.tech/tags/business.md>), [community](<https://devfeed.tech/tags/community.md>), [components](<https://devfeed.tech/tags/components.md>), [designline](<https://devfeed.tech/tags/designline.md>), [digital](<https://devfeed.tech/tags/digital.md>), [education](<https://devfeed.tech/tags/education.md>), [ee-times](<https://devfeed.tech/tags/ee-times.md>), [ee-times-university](<https://devfeed.tech/tags/ee-times-university.md>), [eelife](<https://devfeed.tech/tags/eelife.md>), [eelive-electronic-design](<https://devfeed.tech/tags/eelive-electronic-design.md>), [eet](<https://devfeed.tech/tags/eet.md>), [electrical-engineering](<https://devfeed.tech/tags/electrical-engineering.md>), [electronic-components](<https://devfeed.tech/tags/electronic-components.md>), [electronic-engineering-times](<https://devfeed.tech/tags/electronic-engineering-times.md>), [electronics](<https://devfeed.tech/tags/electronics.md>), [embedded-systems](<https://devfeed.tech/tags/embedded-systems.md>), [first](<https://devfeed.tech/tags/first.md>), [fundamentals-courses](<https://devfeed.tech/tags/fundamentals-courses.md>), [ics](<https://devfeed.tech/tags/ics.md>), [integration](<https://devfeed.tech/tags/integration.md>), [learning](<https://devfeed.tech/tags/learning.md>), [mixed-signal](<https://devfeed.tech/tags/mixed-signal.md>), [motherboard](<https://devfeed.tech/tags/motherboard.md>), [news](<https://devfeed.tech/tags/news.md>), [opinion](<https://devfeed.tech/tags/opinion.md>), [part-search](<https://devfeed.tech/tags/part-search.md>), [passives](<https://devfeed.tech/tags/passives.md>), [product-search](<https://devfeed.tech/tags/product-search.md>), [semiconductors](<https://devfeed.tech/tags/semiconductors.md>), [silicon](<https://devfeed.tech/tags/silicon.md>), [software](<https://devfeed.tech/tags/software.md>), [sponsored-content](<https://devfeed.tech/tags/sponsored-content.md>), [substrate](<https://devfeed.tech/tags/substrate.md>), [tech-papers](<https://devfeed.tech/tags/tech-papers.md>), [technology](<https://devfeed.tech/tags/technology.md>), [times](<https://devfeed.tech/tags/times.md>), [up](<https://devfeed.tech/tags/up.md>), [video](<https://devfeed.tech/tags/video.md>)

### AI overview

Nisshinbo Micro Devices' Silicon Motherboard (Si-MB) packages functional ICs on the top of a silicon substrate and passive components on printed wiring on the bottom, with wafer-formed components inside. The assembly fits existing SOP, QFN, and SON packages. Prototypes reduced mounting area by about 90% for a window comparator and 80% for a current-sense amplifier; the latter uses back-side traces as the resistive element instead of a shunt resistor. The approach combines flexible component placement with dense packaging and is described as more practical for passives than chiplet integration. Existing packages let customers evaluate and prototype without new evaluation equipment. Silicon is rigid, does not absorb moisture, and conducts heat better than organic substrates, though it costs more; the company says it reduces substrate area to control cost and is still working on heat dissipation. It is evaluating systems with manufacturers and targeting automotive, industrial, and AI server applications.

### Source excerpt

Nisshinbo's Silicon Motherboard integrates ICs and passives on both sides of a silicon substrate, reducing mounting area by up to 90%. The post Silicon Motherboard Enables Double-Sided Integration of ICs and Passive Components appeared first on EE Times.

## Robustness Evaluation of GPU Cloud Pricing Benchmarks under Incomplete Observations: A Controlled Missingness Study

DevFeed: [Robustness Evaluation of GPU Cloud Pricing Benchmarks under Incomplete Observations: A Controlled Missingness Study](<https://devfeed.tech/articles/robustness-evaluation-of-gpu-cloud-pricing-benchmarks-under-incomplete-observations-a-controlled-missingness-study-135117.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.17796>)

Author: Jooya Wong

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [reliability-engineering](<https://devfeed.tech/topics/reliability-engineering.md>), [sequential-testing](<https://devfeed.tech/topics/sequential-testing.md>)

Tags: [ablation](<https://devfeed.tech/tags/ablation.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [article](<https://devfeed.tech/tags/article.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [composition](<https://devfeed.tech/tags/composition.md>), [conditional](<https://devfeed.tech/tags/conditional.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [daily](<https://devfeed.tech/tags/daily.md>), [days](<https://devfeed.tech/tags/days.md>), [error](<https://devfeed.tech/tags/error.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [fail](<https://devfeed.tech/tags/fail.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpu-cloud](<https://devfeed.tech/tags/gpu-cloud.md>), [pricing](<https://devfeed.tech/tags/pricing.md>), [robustness](<https://devfeed.tech/tags/robustness.md>), [study](<https://devfeed.tech/tags/study.md>)

### AI overview

The study stress-tests an equal-weight median benchmark for daily GPU rental asking prices under four imposed patterns of missing quotes. Across 5,800 simulations, whole-source outages lowered publication coverage to 0.77 but yielded zero conditional error on dates published; composition-changing quote loss retained 0.86 coverage but caused 7.74 log points of error. A past-only, stratum-level ablation reduced error by 4.15-6.70 log points in five composition-changing cases, using richer information. An observable-signal withholding rule improved the primary error-coverage criterion in none of ten cases. These are controlled results on observed asking prices, not transaction prices or evidence about real-world missingness mechanisms.

### Source excerpt

arXiv:2611.17796v1 Announce Type: new Abstract: Daily GPU rental-price benchmarks can fail when marketplace quotes are missing. This article stress-tests a fixed equal-weight median rule under four imposed masking mechanisms and four benchmark-level recovery methods. Across 5,800 Monte Carlo realisations in 27 scenarios on 190 held-out days, whole-source outages reduce publication coverage to 0.77 while preserving zero conditional error on published dates; composition-changing loss retains 0.86 coverage but produces 7.74 log points of error. A separate past-only stratum-level ablation reduces error by 4.15 to 6.70 log points in five composition-changing cases, using a richer information set. An observable-signal withholding rule improves the primary error-coverage criterion in 0 of 10 cases. Results concern observed asking-price measurements, not transaction prices or identified real-world missingness mechanisms.

## Why Roboflow Sells Hardware, and When to Use Your Own

DevFeed: [Why Roboflow Sells Hardware, and When to Use Your Own](<https://devfeed.tech/articles/why-roboflow-sells-hardware-and-when-to-use-your-own-133570.md>)

Original publisher: [Read original article](<https://roboflow.com/blog/roboflow-hardware-and-when-to-use-your-own>)

Author: Alex Norell

Published: 2026-11-13T23:44:48Z

Content type: article

Language: en

Sources: [Roboflow](<https://devfeed.tech/sources/roboflow-blog.md>)

Topics: [Operating system](<https://devfeed.tech/topics/operating-system.md>), [Embedded Systems](<https://devfeed.tech/topics/embedded-systems.md>), [reliability-engineering](<https://devfeed.tech/topics/reliability-engineering.md>), [on-prem](<https://devfeed.tech/topics/on-prem.md>), [onnx runtime](<https://devfeed.tech/topics/onnx-runtime.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-hardware](<https://devfeed.tech/tags/ai-hardware.md>), [appliances](<https://devfeed.tech/tags/appliances.md>), [compare](<https://devfeed.tech/tags/compare.md>), [drivers](<https://devfeed.tech/tags/drivers.md>), [edge](<https://devfeed.tech/tags/edge.md>), [edge-ai](<https://devfeed.tech/tags/edge-ai.md>), [edge-ai-hardware](<https://devfeed.tech/tags/edge-ai-hardware.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [jetpack](<https://devfeed.tech/tags/jetpack.md>), [patches](<https://devfeed.tech/tags/patches.md>)

### AI overview

Roboflow's appliances bundle hardware with RoboflowOS and five years of over-the-air OS updates, shifting maintenance of the base image, security patches, and JetPack and driver qualification to Roboflow. This addresses host-level issues such as lost network settings after reboot and repairs that otherwise may require shell access or a site visit. Customers managing their own Jetsons still use Deployment Manager for models and device software but must maintain and qualify the OS, drivers, and recovery process; options include their own image and update system or certified Ubuntu images for listed kits. The article recommends keeping managed hosts when teams already handle this work or need shell and kernel-level control, and comparing full-term costs including staff time, support, commissioning, spares, and downtime. Appliances still require production validation of cameras, lighting, and line performance; hardware and camera limitations vary by model.

### Source excerpt

Choosing edge AI hardware means choosing who maintains JetPack, drivers and patches. Compare running your own Jetsons with RoboflowOS appliances.

## Graph-Free Root Cause Analysis

DevFeed: [Graph-Free Root Cause Analysis](<https://devfeed.tech/articles/graph-free-root-cause-analysis-131664.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2601.21359>)

Author: Luan Pham

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [observability](<https://devfeed.tech/topics/observability.md>), [reliability-engineering](<https://devfeed.tech/topics/reliability-engineering.md>)

Tags: [bounded](<https://devfeed.tech/tags/bounded.md>), [complex-systems](<https://devfeed.tech/tags/complex-systems.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [diagnosis](<https://devfeed.tech/tags/diagnosis.md>), [failures](<https://devfeed.tech/tags/failures.md>), [first](<https://devfeed.tech/tags/first.md>), [framework](<https://devfeed.tech/tags/framework.md>), [graph](<https://devfeed.tech/tags/graph.md>), [latency](<https://devfeed.tech/tags/latency.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [root-cause-analysis](<https://devfeed.tech/tags/root-cause-analysis.md>)

### AI overview

CIE ranks root causes by combining internal anomalies, such as resource usage, with external anomalies, such as latency. This addresses a weakness of graph-free methods that rank by anomaly magnitude alone: fault propagation can make downstream deviations larger than the root fault. The paper states that a conjunctive score provably ranks the root first under arbitrary amplification; its default additive score requires bounded amplification and a stronger root-cause condition. Across 735 failures in nine datasets from three microservice systems, CIE reached 68% Top-1 accuracy averaged over 11 fault types, a 21% relative improvement over the best of 12 baselines, at 8 ms per diagnosis.

### Source excerpt

arXiv:2601.21359v2 Announce Type: replace-cross Abstract: Failures in complex systems demand rapid Root Cause Analysis (RCA) to prevent cascading damage. Graph-free RCA methods typically assume that the root cause has the highest anomaly score. This assumption fails when faults propagate, because a small deviation at the root cause can grow into a larger anomaly downstream. Our key observation is that a root cause deviates in both its internal properties (e.g., resource usage) and external properties (e.g., latency), whereas components affected by propagation deviate only in external properties. We propose CIE, a framework that ranks components by combining their internal and external anomaly scores. We formalize a class of component-based systems and state conditions under which CIE provably ranks the root cause first. The guarantee holds under arbitrary amplification for a conjunctive score, and under bounded amplification and a stronger root-cause condition for the default additive score. On 735 failures in nine datasets from three microservice systems, CIE achieves 68% Top-1 accuracy averaged over 11 fault types, a 21% relative improvement over the best of twelve baselines, while requiring 8 ms per diagnosis.

## Calibrated selective prediction using deep ensembles for thyroid nodule ultrasound classification under dataset shift

DevFeed: [Calibrated selective prediction using deep ensembles for thyroid nodule ultrasound classification under dataset shift](<https://devfeed.tech/articles/calibrated-selective-prediction-using-deep-ensembles-for-thyroid-nodule-ultrasound-classification-under-dataset-shift-131950.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2607.12075>)

Author: Md. Sadibul Hasan Sadib, Md. Mohayminul Mukit, Rahmatul Kabir Rasel Sarker, Tahmid Alam Tamim, Md. Nazmul Hasan, Md. Monir Hossain Shimul

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Calibrated uncertainty](<https://devfeed.tech/topics/calibrated-uncertainty.md>), [Image processing](<https://devfeed.tech/topics/image-processing.md>), [reliability-engineering](<https://devfeed.tech/topics/reliability-engineering.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>)

Tags: [classification](<https://devfeed.tech/tags/classification.md>), [cross-validation](<https://devfeed.tech/tags/cross-validation.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [deep](<https://devfeed.tech/tags/deep.md>), [deep-learning](<https://devfeed.tech/tags/deep-learning.md>), [eess-iv](<https://devfeed.tech/tags/eess-iv.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [fna](<https://devfeed.tech/tags/fna.md>), [framework](<https://devfeed.tech/tags/framework.md>), [image](<https://devfeed.tech/tags/image.md>), [msp](<https://devfeed.tech/tags/msp.md>), [performance](<https://devfeed.tech/tags/performance.md>), [policy](<https://devfeed.tech/tags/policy.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [risk](<https://devfeed.tech/tags/risk.md>), [roc](<https://devfeed.tech/tags/roc.md>), [selection](<https://devfeed.tech/tags/selection.md>), [shift](<https://devfeed.tech/tags/shift.md>), [site](<https://devfeed.tech/tags/site.md>), [test](<https://devfeed.tech/tags/test.md>), [ultrasound](<https://devfeed.tech/tags/ultrasound.md>)

### AI overview

A five-member ConvNeXt-Tiny ensemble with uncertainty-aware selective triage performed well on TN5000 but showed substantial dataset-shift degradation on TN3K, limiting its transportability. On TN5000, out-of-fold AUC-ROC was 0.9377 and expected calibration error 0.0138. The primary mutual-information policy assigned 8.86% of images a No-FNA suggestion, 60.00% an FNA recommendation, and 31.14% for radiologist review; 0.39% of malignant images received No-FNA. Predictive variance and maximum-softmax probability ranked errors better than mutual information, while no uncertainty measure optimized every triage outcome. Applied to TN3K without recalibration, AUC-ROC fell to 0.7870, calibration error rose to 0.1899, and 83.7% of images were sent for review. The authors call for multicentre validation, site-specific recalibration, threshold validation, and prospective evaluation before clinical use.

### Source excerpt

arXiv:2607.12075v3 Announce Type: replace-cross Abstract: Background: Deep learning can classify thyroid nodules on ultrasound, but discrimination alone does not establish clinical reliability. We developed a calibrated deep-ensemble framework with uncertainty-aware selective prediction and conservative image-based triage. Methods: Nested five-fold cross-validation with three-fold inner architecture selection chose ConvNeXt-Tiny with a squeeze-and-excitation block, used to build a deterministic five-member ensemble. Training, early-stopping, calibration, and threshold subsets were separated within each fold; outer-test data remained isolated. Mutual information (MI), predictive variance, and maximum-softmax-probability (MSP) uncertainty were compared by risk-coverage and triage analyses, and predictive values across assumed malignancy prevalences of 5%-70%. The frozen TN5000 ensemble was applied to TN3K without recalibration. Results: On TN5000, pooled out-of-fold AUC-ROC was 0.9377, average precision 0.9708, expected calibration error 0.0138, and Brier score 0.0827. Under the primary MI-guided policy, 8.86% of images received a No-FNA suggestion, 60.00% an FNA recommendation, and 31.14% radiologist review; of 3574 malignant images, 14 (0.39%) received No-FNA and 708 (19.81%) review. Predictive variance and MSP ranked errors better than MI (AURC, AUGRC); no measure optimized all triage outcomes. Across prevalences of 5%-70%, NPV ranged 99.95%-97.93% and PPV 29.28%-94.75%. On TN3K, AUC-ROC was 0.7870, average precision 0.7254, expected calibration error 0.1899, and radiologist-review rate 83.7%. Conclusion: The ensemble showed strong internal discrimination and calibration, but performance deteriorated substantially on TN3K, indicating limited transportability under dataset shift. Multicentre validation, site-specific recalibration, threshold validation, and prospective evaluation are needed before clinical use.

## When Failures Propagate: Causal Failure Attribution in Agentic Retrieval-Augmented Generation

DevFeed: [When Failures Propagate: Causal Failure Attribution in Agentic Retrieval-Augmented Generation](<https://devfeed.tech/articles/when-failures-propagate-causal-failure-attribution-in-agentic-retrieval-augmented-generation-131959.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2608.20627>)

Author: Lauren Pothuru, Spurthi Setty, Natan Vidra

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>), [causality](<https://devfeed.tech/topics/causality.md>), [reliability-engineering](<https://devfeed.tech/topics/reliability-engineering.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-rag](<https://devfeed.tech/tags/agentic-rag.md>), [agentic-retrieval](<https://devfeed.tech/tags/agentic-retrieval.md>), [attribution](<https://devfeed.tech/tags/attribution.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [claude](<https://devfeed.tech/tags/claude.md>), [claude-haiku](<https://devfeed.tech/tags/claude-haiku.md>), [claude-haiku-4-5](<https://devfeed.tech/tags/claude-haiku-4-5.md>), [corruption](<https://devfeed.tech/tags/corruption.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [diagnosis](<https://devfeed.tech/tags/diagnosis.md>), [error](<https://devfeed.tech/tags/error.md>), [failure](<https://devfeed.tech/tags/failure.md>), [failures](<https://devfeed.tech/tags/failures.md>), [generation](<https://devfeed.tech/tags/generation.md>), [haiku](<https://devfeed.tech/tags/haiku.md>), [haiku-4-5](<https://devfeed.tech/tags/haiku-4-5.md>), [llm](<https://devfeed.tech/tags/llm.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [retrieval-augmented-generation](<https://devfeed.tech/tags/retrieval-augmented-generation.md>)

### AI overview

AgenticRAG-FP tests whether a diagnoser can identify the hop where a failure was injected after downstream execution changes. It injects a certified fault at a known hop, re-executes the remaining trajectory, and scores diagnoses against that intervention site. In a dense Claude Haiku 4.5 sweep on three-hop MuSiQue questions, coverage-based diagnosis achieved 0.91 accuracy at hop 1 but 0.00 at hops 2 and 3, while predicting hop 1 for 96% of failures. For later-hop failures, an LLM judge and frozen-hop repair localized only 0.25-0.48. In larger depth-2 content-corruption runs (n=51), exact-hop accuracy was 0.00 for coverage, 0.49 for the judge, 0.59 for frozen-hop repair, and 0.14 for suffix regeneration. The judge and frozen-hop repair were not significantly different (exact McNemar p=0.33).

### Source excerpt

arXiv:2608.20627v2 Announce Type: replace-cross Abstract: Agentic retrieval-augmented generation (RAG) interleaves retrieval, reasoning, and answer generation across multiple hops. A retrieval error at hop 1 can surface only as a wrong answer at hop 3, while later retrieval can also repair the trajectory. This paper introduces AgenticRAG-FP, an interventional benchmark for causal failure attribution in agentic RAG. The benchmark injects a certified fault at a specified hop, re-executes the downstream trajectory, and evaluates diagnosers against the known intervention site. Its central question is whether a post-hoc trace still identifies the injected hop after the suffix changes. In the dense Claude Haiku 4.5 structural sweep on three-hop MuSiQue questions, coverage-based diagnosis is 0.91 at hop 1 and 0.00 at hops 2 and 3 (n=43,36,21 failed trajectories), but it predicts hop 1 in 96% of failures, an earliest-hop rule; at hops 2 and 3, an LLM judge and frozen-hop repair localize only 0.25 to 0.48 of failures. In larger content-corruption runs, depth-2 exact-hop accuracy is 0.00 for coverage, 0.49 for an LLM judge, 0.59 for frozen-hop repair, and 0.14 for suffix regeneration (n=51); the judge and frozen-hop repair do not significantly differ (exact McNemar p=0.33).

## FIRE: Failure Isolation and Reliability for Production-Connected Experiments at Scale

DevFeed: [FIRE: Failure Isolation and Reliability for Production-Connected Experiments at Scale](<https://devfeed.tech/articles/fire-failure-isolation-and-reliability-for-production-connected-experiments-at-scale-131527.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.14962>)

Author: Shripad Nadgowda, Gaurav Mitra, Sing Sing Ma, Vipul Chaskar, Pradipta Ghosh, Canturk Isci, Onkar Deshpande, Edgardo Aviles-Lopez, Ryan Feng, J\\'an Ondru\\v{s}ek

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [reliability-engineering](<https://devfeed.tech/topics/reliability-engineering.md>), [Production Engineering](<https://devfeed.tech/topics/production-engineering.md>), [product-experimentation](<https://devfeed.tech/topics/product-experimentation.md>), [staging](<https://devfeed.tech/topics/staging.md>)

Tags: [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [dependencies](<https://devfeed.tech/tags/dependencies.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [failure](<https://devfeed.tech/tags/failure.md>), [guardrails](<https://devfeed.tech/tags/guardrails.md>), [isolation](<https://devfeed.tech/tags/isolation.md>), [monitors](<https://devfeed.tech/tags/monitors.md>), [production](<https://devfeed.tech/tags/production.md>), [ranking](<https://devfeed.tech/tags/ranking.md>), [recommendation](<https://devfeed.tech/tags/recommendation.md>), [record-and-replay](<https://devfeed.tech/tags/record-and-replay.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [scale](<https://devfeed.tech/tags/scale.md>)

### AI overview

FIRE supports production-connected experiments at scale by combining three guardrails: isolated tiers redirect replay traffic away from shared dependencies where replication is practical; dependency monitors track downstream health and experiment-attributed errors, stopping experiments after sustained violations; and traffic pinning confines calls to selected downstream workers when full replication would use scarce capacity. In a large-scale recommendation system, isolated ranking tiers handled billions of replayed requests daily with under 1% of corresponding serving capacity. Across 90 days, more than 200,000 abnormal worker exits in those tiers did not directly affect production-serving workers. In a separate three-month period, monitors cancelled about 5% of experiments for the measured ranking service and accounted for 87% of its cancellations. Dependency-aware triggering reduced code-review experiment executions by 25%.

### Source excerpt

arXiv:2611.14962v1 Announce Type: cross Abstract: Pre-production changes are usually tested in isolated environments to prevent their side effects from reaching production. Some record-and-replay platforms allow a production-connected mode: candidate changes receive recorded production requests while selected downstream calls reach production dependencies. This improves fidelity before deployment, but it also introduces reliability risks. At hyperscale, experimental traffic can fan out across shared services, trigger latent defects, or send malformed input to systems serving user requests. In this paper, we show how production-connected experimentation can operate reliably in a large-scale recommendation serving system. To contain downstream effects without sacrificing realistic dependency behavior, the platform uses three complementary guardrails. Isolated tiers redirect replay traffic away from shared dependencies where replication is practical. Dependency monitors observe downstream health and experiment-attributed errors and terminate experiments after sustained violations. Traffic pinning limits experimental calls to a designated subset of downstream workers when full replication would require scarce capacity. Isolated ranking tiers process billions of replayed requests per day using under 1% of the corresponding serving capacity. Over 90 days, these tiers experienced more than 200,000 abnormal worker exits without directly affecting production-serving workers. During a separate three-month window, dependency monitors cancelled approximately 5% of experiments for the measured ranking service and accounted for 87% of its cancellations. Dependency-aware triggering also reduced code-review experiment executions by 25%. These results show that high-fidelity, production-connected experimentation can operate at scale when downstream effects are isolated where possible, bounded where necessary, and continuously monitored and attributed.

## How to Benchmark LLMs

DevFeed: [How to Benchmark LLMs](<https://devfeed.tech/articles/how-to-benchmark-llms-129838.md>)

Original publisher: [Read original article](<https://openrouter.ai/blog/tutorials/how-to-benchmark-llms/>)

Author: OpenRouter

Published: 2026-11-12T00:00:00Z

Content type: tutorial

Language: en

Sources: [OpenRouter](<https://devfeed.tech/sources/openrouter.md>)

Topics: [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [test](<https://devfeed.tech/topics/test.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>), [reliability-engineering](<https://devfeed.tech/topics/reliability-engineering.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [compare](<https://devfeed.tech/tags/compare.md>), [cost](<https://devfeed.tech/tags/cost.md>), [gpt](<https://devfeed.tech/tags/gpt.md>), [gpt-6-1](<https://devfeed.tech/tags/gpt-6-1.md>), [gpt-6-1-sol](<https://devfeed.tech/tags/gpt-6-1-sol.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [llms](<https://devfeed.tech/tags/llms.md>), [models](<https://devfeed.tech/tags/models.md>), [openrouter](<https://devfeed.tech/tags/openrouter.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>)

### AI overview

Benchmark candidate models on representative tasks with independently computed answers and code that checks both correctness and output format. Validate the grader with boundary, malformed, and known-wrong cases; run identical tasks under fixed settings; track response coverage, verified cost, and request time; and retain repeats and unresolved attempts. In this 60-task synthetic study, GPT-6.1 Sol passed 180/180 trials at about $0.2449 per 100 responses, while DeepSeek had the lowest cost (about $0.00394) with 157/180 passes and 14 truncations. Sonnet's many failures included malformed outputs, so scores reflect contract compliance as well as correctness. Public ranking agreement was weak in this ten-model sample (Spearman 0.109). These results apply only to the tested tasks and settings; the text recommends held-out cases and more repeats for close rankings.

### Source excerpt

Choose models using evidence from your application's inputs and output formats. Run identical tasks through OpenRouter and compare graded accuracy, billed cost, and request time. On our 60 synthetic tasks, GPT-6.1 Sol passed 180/180 trials across three repeats under the tested contracts and settings.

## TWIF: November 02 to 08, 2026

DevFeed: [TWIF: November 02 to 08, 2026](<https://devfeed.tech/articles/twif-november-02-to-08-2026-129352.md>)

Original publisher: [Read original article](<https://communityblog.fedoraproject.org/this-week-in-fedora-2026-11-02/>)

Author: Justin Wheeler

Published: 2026-11-11T17:02:45Z

Content type: news

Language: en

Sources: [Fedora Community Blog](<https://devfeed.tech/sources/fedora-community-blog.md>)

Topics: [Fedora](<https://devfeed.tech/topics/fedora.md>), [Fedora Project Community](<https://devfeed.tech/topics/fedora-project-community.md>), [release engineering](<https://devfeed.tech/topics/release-engineering.md>), [Test-driven development](<https://devfeed.tech/topics/tdd.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>), [reliability-engineering](<https://devfeed.tech/topics/reliability-engineering.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>)

Tags: [2020s](<https://devfeed.tech/tags/2020s.md>), [2026](<https://devfeed.tech/tags/2026.md>), [ai-assistance-used](<https://devfeed.tech/tags/ai-assistance-used.md>), [ambassadors](<https://devfeed.tech/tags/ambassadors.md>), [blog](<https://devfeed.tech/tags/blog.md>), [community](<https://devfeed.tech/tags/community.md>), [copr](<https://devfeed.tech/tags/copr.md>), [coreos](<https://devfeed.tech/tags/coreos.md>), [council](<https://devfeed.tech/tags/council.md>), [design](<https://devfeed.tech/tags/design.md>), [development-packaging](<https://devfeed.tech/tags/development-packaging.md>), [documentation](<https://devfeed.tech/tags/documentation.md>), [ecosystem](<https://devfeed.tech/tags/ecosystem.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [epel](<https://devfeed.tech/tags/epel.md>), [fedora](<https://devfeed.tech/tags/fedora.md>), [fedora-atomic](<https://devfeed.tech/tags/fedora-atomic.md>), [fedora-kde](<https://devfeed.tech/tags/fedora-kde.md>), [fedora-linux](<https://devfeed.tech/tags/fedora-linux.md>), [fedora-linux-45](<https://devfeed.tech/tags/fedora-linux-45.md>), [gnome](<https://devfeed.tech/tags/gnome.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [iot](<https://devfeed.tech/tags/iot.md>), [join](<https://devfeed.tech/tags/join.md>), [python](<https://devfeed.tech/tags/python.md>), [quality-assurance-qa](<https://devfeed.tech/tags/quality-assurance-qa.md>), [release-engineering](<https://devfeed.tech/tags/release-engineering.md>), [risc-v](<https://devfeed.tech/tags/risc-v.md>), [rust](<https://devfeed.tech/tags/rust.md>), [server](<https://devfeed.tech/tags/server.md>), [this-week-in-fedora](<https://devfeed.tech/tags/this-week-in-fedora.md>), [websites-apps](<https://devfeed.tech/tags/websites-apps.md>), [workstation](<https://devfeed.tech/tags/workstation.md>)

### AI overview

Fedora Linux 45 was released, and teams moved into Fedora 46 planning. The release brought GNOME 51, KDE Plasma 6.8, Apple M3 support in Fedora Asahi Remix, and native Stratis configuration in Anaconda; the unmaintained NetworkManager VPN plugins were retired. Work also centered on bootable image unification, while Fedora groups weighed a possible Bugzilla-to-Forgejo migration and completed DNF4 removal from ELN. Teams addressed security updates and infrastructure issues, including shipping a CoreOS fix for an IPv4 TCP use-after-free vulnerability. The report also notes release-specific tradeoffs and active needs: Plasma 6.8 drops X11 sessions, the HCP Vagrant Registry is scheduled to shut down December 31, and contributors are sought for testing, documentation, package security updates, and infrastructure work. This overview reflects the supplied text.

### Source excerpt

Across the Fedora ecosystem this week, the primary focus was the highly anticipated release of Fedora Linux 45, with teams including Release Engineering... The post TWIF: November 02 to 08, 2026 appeared first on Fedora Community Blog.

## Darmok: Championship Cryptic Crossword Solving via Constraint Tightening and Resilient Orchestration

DevFeed: [Darmok: Championship Cryptic Crossword Solving via Constraint Tightening and Resilient Orchestration](<https://devfeed.tech/articles/darmok-championship-cryptic-crossword-solving-via-constraint-tightening-and-resilient-orchestration-127808.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.13557>)

Author: Siddhartha Chatterjee (The University of Texas at Austin)

Published: 2026-11-11T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [AI Architecture](<https://devfeed.tech/topics/ai-architecture.md>), [Frontier AI](<https://devfeed.tech/topics/frontier-ai.md>), [reliability-engineering](<https://devfeed.tech/topics/reliability-engineering.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-systems](<https://devfeed.tech/tags/ai-systems.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [crosswords](<https://devfeed.tech/tags/crosswords.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [orchestration](<https://devfeed.tech/tags/orchestration.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [resilience](<https://devfeed.tech/tags/resilience.md>)

### AI overview

Darmok solved all seven puzzles at the 55th Times Crossword Championship without errors, finishing faster than every human competitor in each round; its Final time was 6m 52s, 3m 13s ahead of the human champion. The system combines rounds of clue cross-checking across three independent models with verified crossing letters, then uses a whole-puzzle model and dictionary lookups to fill gaps without changing confirmed words. This orchestration maintained near-zero-error performance over five months of daily evaluation. Although removing backups and fallbacks barely affected ordinary benchmark accuracy, those paths proved important on difficult operating days and during a severe slowdown of the primary model in the Final. The abstract argues that resilience to worst-case failures becomes central as models handle average cases well.

### Source excerpt

arXiv:2611.13557v1 Announce Type: cross Abstract: We present Darmok, the first automated system to reach superhuman performance on British cryptic crosswords. Unlike American-style crosswords, cryptic clues disguise an unmarked definition and compositional wordplay beneath a misleading surface reading, sparse grids leave nearly half of each word's letters unchecked, and tournament rules rank a grid with even a single wrong or blank square behind all error-free solutions regardless of speed. Competing alongside an elite human field at the 55th Times Crossword Championship in London, Darmok solved all seven championship puzzles error-free and finished faster than every human competitor in every round. It solved the Final puzzle in 6m 52s, a full 3m 13s ahead of the human champion. Today's frontier AI models can on average solve a whole puzzle in a single try with comparable accuracy and speed. However, cloud API calls still stall, time out, trigger safety refusals, or get stuck in reasoning loops. To simultaneously achieve tournament-grade accuracy, speed, and reliability, Darmok cross-checks clues across three independent models in rounds; uses verified crossing letters to unlock harder clues; and uses a whole-puzzle model and dictionary lookups to fill any remaining gaps without overwriting confirmed words. This design maintained near-zero-error performance across a five-month daily evaluation period. Under normal benchmark conditions, removing backup models, fallback steps, or elaborate prompt instructions barely changes accuracy, making the architecture appear over-engineered. Yet during both the worst days of daily operation and a severe slowdown of our primary model in the live Championship Final, the backup paths carried the solve, showing that as foundation models master the average case, the central challenge in building complex AI systems shifts to resilience in the worst case.

[Next page](<https://devfeed.tech/topics/reliability-engineering.md?cursor=WyIyMDI2LTExLTExVDA1OjAwOjAwKzAwOjAwIiwgImZmZGVjOTFjLWNhODMtNGNjYi1hODU0LTgwODFiYjYwM2E0MSJd>)