# When the bug slips through: How we built an AI feedback loop to strengthen our safety net DevFeed: [When the bug slips through: How we built an AI feedback loop to strengthen our safety net]() Original publisher: [Read original article]() Author: Vasilescu Andreea Published: 2026-07-14T07:16:00Z Content type: article Language: en Sources: [Data Science at Microsoft]() Topics: [Artificial Intelligence](), [bug](), [incident](), [Monitoring](), [Software](), [data](), [Azure]() Tags: [ai](), [analysis](), [artificial-intelligence](), [automation](), [bug](), [data-science](), [engineering](), [errors](), [incident](), [incident-management](), [issue](), [llm](), [monitoring](), [outage](), [production](), [software](), [teams]() ## AI overview The article describes how a large-scale engineering organization built an AI feedback loop to investigate production regressions, compare them with pre-production alerts, and identify gaps in its quality platform. It explains that manual incident sampling was valuable but covered less than 10 percent of incidents, making broader forensic analysis impractical at Azure scale. ## Source excerpt Figure 1: When the bug slips through, or how we built an AI feedback loop to strengthen our safety net. A regression reached production last fall. Not dramatically, no war-room scramble, no cascading outage. It showed up as a cluster of customer-facing errors, traced back through monitoring data to a change that had passed through every pre-production check without triggering a single alert. Somewhere upstream, the safety net had a hole that nobody knew was there. This scenario is not hypothetical. It happens. And when it does, the instinctive response is the same: investigate the incident, fix the issue, and then ask the uncomfortable follow-up question, Why didn't we catch this before it reached production? Answering that question is deceptively hard. A pre-production quality platform generates a constant stream of signals, warnings, and anomalies, many of them noisy, overlapping, or ultimately irrelevant. The regression in question might be buried among production incidents, false signals, and failures the platform was never designed to catch in the first place. While manual sampling and investigation can provide meaningful insight, understanding what was genuinely missed, why it was missed, and what needs to change at platform scale requires the kind of careful forensic analysis that is slow, expert-intensive, and -- at the scale of Azure -- practically impossible to do by hand. This is the problem we set out to solve. A safety net with blind spots Modern large-scale engineering organizations invest heavily in pre-production quality platforms, systems that evaluate software changes before they reach customers. These are systems designed to catch regressions before any code change reaches customers, running controlled experiments, executing targeted tests, monitoring key signals, and alerting engineering teams when something looks wrong. But a safety net is only as good as its coverage. And knowing how good your coverage actually is requires a feedback mechanism: a