# GCP outage lessons learned

Lessons learned from handling a Google Cloud Platform outage, including preparing for failures, progressively fixing incidents, and preventing recurrence.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## 'Too big to not fail': Google outage exposes UK reliance on a handful of cloud providers

DevFeed: ['Too big to not fail': Google outage exposes UK reliance on a handful of cloud providers](<https://devfeed.tech/articles/too-big-to-not-fail-google-outage-exposes-uk-reliance-on-a-handful-of-cloud-providers-90950.md>)

Original publisher: [Read original article](<https://www.theguardian.com/uk-news/2026/oct/20/power-outage-google-datacentre-london-widespread-disruption>)

Author: Aisha Down

Published: 2026-10-20T16:50:31Z

Content type: news

Language: en

Sources: [The Guardian | Technology](<https://devfeed.tech/sources/the-guardian-technology.md>)

Topics: [Google Cloud Platform (GCP)](<https://devfeed.tech/topics/google-cloud.md>), [GCP outage lessons learned](<https://devfeed.tech/topics/gcp-outage-lessons-learned.md>), [Cloud](<https://devfeed.tech/topics/cloud.md>), [Infrastructure as a Service (IaaS)](<https://devfeed.tech/topics/infrastructure-as-a-service-iaas.md>), [amazon](<https://devfeed.tech/topics/amazon.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [amazon-web-services](<https://devfeed.tech/tags/amazon-web-services.md>), [app](<https://devfeed.tech/tags/app.md>), [azure](<https://devfeed.tech/tags/azure.md>), [banking](<https://devfeed.tech/tags/banking.md>), [business](<https://devfeed.tech/tags/business.md>), [cascade](<https://devfeed.tech/tags/cascade.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-services](<https://devfeed.tech/tags/cloud-services.md>), [court](<https://devfeed.tech/tags/court.md>), [data](<https://devfeed.tech/tags/data.md>), [datacentres](<https://devfeed.tech/tags/datacentres.md>), [datacentres-uk](<https://devfeed.tech/tags/datacentres-uk.md>), [datacentres-uk-uk-news-technology-google](<https://devfeed.tech/tags/datacentres-uk-uk-news-technology-google.md>), [europe](<https://devfeed.tech/tags/europe.md>), [google](<https://devfeed.tech/tags/google.md>), [hyperscalers](<https://devfeed.tech/tags/hyperscalers.md>), [internet](<https://devfeed.tech/tags/internet.md>), [london](<https://devfeed.tech/tags/london.md>), [outage](<https://devfeed.tech/tags/outage.md>), [power](<https://devfeed.tech/tags/power.md>), [power-outage](<https://devfeed.tech/tags/power-outage.md>), [technology](<https://devfeed.tech/tags/technology.md>), [uk](<https://devfeed.tech/tags/uk.md>), [uk-news](<https://devfeed.tech/tags/uk-news.md>)

### AI overview

A power event at a third-party facility in Google Cloud's europe-west2 region disrupted services used by UK banks, courts and the NHS app; Google said services had been restored and promised an incident analysis. The reported effects were narrower than the previous year's roughly 15-hour AWS outage, which affected services worldwide. Internet infrastructure expert Michał Woźniak says reliance on a few hyperscalers leaves public institutions and large parts of the economy exposed to cascading failures. The excerpt also notes that a separate AWS outage was linked to a bug in automation software and another to an AI tool, and raises questions about reliability as Google adopts AI-generated code. It provides no evidence that AI caused this Google outage.

### Source excerpt

Our dependence on internet services of a few key tech companies leaves us vulnerable to cloud failures, bugs and more, experts say It happened again. A power outage hit a datacentre in London, and things cascaded from there: multiple banking services faltered, court hearings were put on hold and patients reported temporary glitches on the NHS app. The datacentre in question was in Google's europe-west2-b zone, one of more than a dozen "zones" that the US search giant maintains in Europe to service its clients. Continue reading...

## Coveralls outage followed an AWS account shutdown

DevFeed: [Coveralls outage followed an AWS account shutdown](<https://devfeed.tech/articles/the-outage-90637.md>)

Original publisher: [Read original article](<https://thedailywtf.com/articles/the-outage>)

Author: Remy Porter

Published: 2026-10-20T06:30:00Z

Content type: opinion

Language: en

Sources: [The Daily WTF](<https://devfeed.tech/sources/the-daily-wtf.md>)

Topics: [GCP outage lessons learned](<https://devfeed.tech/topics/gcp-outage-lessons-learned.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>), [hosting](<https://devfeed.tech/topics/hosting.md>)

Tags: [aws](<https://devfeed.tech/tags/aws.md>), [ci](<https://devfeed.tech/tags/ci.md>), [feature-articles](<https://devfeed.tech/tags/feature-articles.md>), [outage](<https://devfeed.tech/tags/outage.md>), [post-mortem](<https://devfeed.tech/tags/post-mortem.md>)

### AI overview

Coveralls' outage caused CI jobs using its code coverage reporting service to fail. The author says AWS shut down the service after mistakenly deciding Coveralls had not paid its bills, and that it took about 68 hours to identify the issue. The piece humorously reads excerpts from Coveralls' outage updates.

### Source excerpt

At my employer, we use a service called Coveralls, which is a code coverage reporting tool that hooks into your CI pipeline. It takes your lcov reports and gives you some insights, and also makes it easy to cause the CI pipeline to fail if newly added code doesn't have reasonable coverage. I have no strong feelings about the tool, but on February 24th, our CI jobs started failing. Coveralls had gone down, and our CI step that talked to them now failed. That happened to us, and a lot of other people. Coveralls actively published updates as the outage dragged on, and boy, did it drag on. Between them posting nearly identical updates minutes or even seconds apart and the general desperation that snuck into their outage notes after the first few days, I couldn't help but find their notes hilarious. They've published their post-mortem, and the short version is that despite them paying their bills, Amazon decided they hadn't, and shut down their services. But it took them roughly 68 hours to figure that out. If you scroll down past the post-mortem, you can see their outage notes. This is an unauthorized, dramatic reading of those notes. We've taken some liberties in interpreting it. I hope this is taken in a good spirit, because they were just doing their best when this happened. I don't think most companies would be able to maintain operations when their cloud provider decides to lock their accounts. But I don't care who you are: begging your users to email you if they can help you migrate off AWS is funny. [Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!

## Yandex confirms service outage as users lose access to cloud files

DevFeed: [Yandex confirms service outage as users lose access to cloud files](<https://devfeed.tech/articles/yandex-confirms-service-outage-as-users-lose-access-to-cloud-files-85643.md>)

Original publisher: [Read original article](<https://www.bleepingcomputer.com/news/security/yandex-confirms-service-outage-as-users-lose-access-to-cloud-files/>)

Author: Bill Toulas

Published: 2026-10-12T19:23:01Z

Content type: news

Language: en

Sources: [BleepingComputer](<https://devfeed.tech/sources/bleepingcomputer.md>)

Topics: [cloud service outage management](<https://devfeed.tech/topics/cloud-service-outage-management.md>), [data centers](<https://devfeed.tech/topics/data-centers.md>), [GCP outage lessons learned](<https://devfeed.tech/topics/gcp-outage-lessons-learned.md>), [Infrastructure as a Service (IaaS)](<https://devfeed.tech/topics/infrastructure-as-a-service-iaas.md>)

Tags: [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-infrastructure-services](<https://devfeed.tech/tags/cloud-infrastructure-services.md>), [computer-help](<https://devfeed.tech/tags/computer-help.md>), [computer-security](<https://devfeed.tech/tags/computer-security.md>), [computers](<https://devfeed.tech/tags/computers.md>), [data-centers](<https://devfeed.tech/tags/data-centers.md>), [disaster](<https://devfeed.tech/tags/disaster.md>), [disaster-recovery](<https://devfeed.tech/tags/disaster-recovery.md>), [infosec](<https://devfeed.tech/tags/infosec.md>), [infosec-computer-security](<https://devfeed.tech/tags/infosec-computer-security.md>), [it](<https://devfeed.tech/tags/it.md>), [it-outage](<https://devfeed.tech/tags/it-outage.md>), [linux](<https://devfeed.tech/tags/linux.md>), [mac](<https://devfeed.tech/tags/mac.md>), [malware](<https://devfeed.tech/tags/malware.md>), [malware-removal](<https://devfeed.tech/tags/malware-removal.md>), [outage](<https://devfeed.tech/tags/outage.md>), [russia](<https://devfeed.tech/tags/russia.md>), [security](<https://devfeed.tech/tags/security.md>), [spyware](<https://devfeed.tech/tags/spyware.md>), [support](<https://devfeed.tech/tags/support.md>), [tech-support](<https://devfeed.tech/tags/tech-support.md>), [technical-support](<https://devfeed.tech/tags/technical-support.md>), [ukraine](<https://devfeed.tech/tags/ukraine.md>), [virus](<https://devfeed.tech/tags/virus.md>), [virus-removal](<https://devfeed.tech/tags/virus-removal.md>), [windows](<https://devfeed.tech/tags/windows.md>), [yandex](<https://devfeed.tech/tags/yandex.md>)

### AI overview

Drone attacks damaged three Yandex data centers, disrupting cloud services and leaving some customers unable to access stored files. Yandex recommended that customers use alternative disaster recovery plans and said it could not provide a reliable restoration timeline. The status update described varying availability across services, while the extent of any permanent data loss remained unclear.

### Source excerpt

Russian cloud services provider Yandex announced that drone attacks damaged three of its data centers, forcing it to shut down two facilities and partially disable a third. [...]

## The Pulse: Firebase's global outage & poor response

DevFeed: [The Pulse: Firebase's global outage & poor response](<https://devfeed.tech/articles/the-pulse-firebase-s-global-outage-poor-response-77457.md>)

Original publisher: [Read original article](<https://blog.pragmaticengineer.com/the-pulse-firebases-global-outage-poor-response/>)

Author: Ivan Klaric

Published: 2026-10-08T16:56:37Z

Content type: news

Language: en

Sources: [The Pragmatic Engineer](<https://devfeed.tech/sources/the-pragmatic-engineer-2.md>)

Topics: [cloud service outage management](<https://devfeed.tech/topics/cloud-service-outage-management.md>), [incident](<https://devfeed.tech/topics/incident.md>), [Firebase](<https://devfeed.tech/topics/firebase.md>), [GCP outage lessons learned](<https://devfeed.tech/topics/gcp-outage-lessons-learned.md>), [DORA change failure rate](<https://devfeed.tech/topics/dora-change-failure-rate.md>)

Tags: [firebase](<https://devfeed.tech/tags/firebase.md>), [ios](<https://devfeed.tech/tags/ios.md>), [outage](<https://devfeed.tech/tags/outage.md>), [postmortem](<https://devfeed.tech/tags/postmortem.md>), [sdk](<https://devfeed.tech/tags/sdk.md>), [status](<https://devfeed.tech/tags/status.md>)

### AI overview

A backend configuration change caused Firebase Analytics iOS SDK crashes that affected apps for up to six hours. The article examines the outage response, including delayed acknowledgment and status page updates, and describes Google's later postmortem and planned dashboard improvements. It also recommends testing mobile apps against malformed configuration payloads.

### Source excerpt

The Firebase iOS SDK crashed after a backend change, crashing all apps which used Firebase analytics for 2-6 hours. Google didn't update the status page, but did offer a postmortem 4 days later.

## Google overhauls cloud modernization portfolio with new AI agents

DevFeed: [Google overhauls cloud modernization portfolio with new AI agents](<https://devfeed.tech/articles/google-overhauls-cloud-modernization-portfolio-with-new-ai-agents-65468.md>)

Original publisher: [Read original article](<https://www.networkworld.com/article/4231147/google-overhauls-cloud-modernization-portfolio-with-new-ai-agents.html>)

Author: Anirban Ghoshal

Published: 2026-10-06T12:08:21Z

Content type: news

Language: en

Sources: [Network World](<https://devfeed.tech/sources/network-world.md>)

Topics: [GCP outage lessons learned](<https://devfeed.tech/topics/gcp-outage-lessons-learned.md>), [Google Cloud Platform (GCP)](<https://devfeed.tech/topics/google-cloud.md>), [Gke](<https://devfeed.tech/topics/gke.md>), [Loop Engineering](<https://devfeed.tech/topics/loop-engineering.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [amazon-eks](<https://devfeed.tech/tags/amazon-eks.md>), [application-modernization](<https://devfeed.tech/tags/application-modernization.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [artificial-intelligence-cloud-computing](<https://devfeed.tech/tags/artificial-intelligence-cloud-computing.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-computing](<https://devfeed.tech/tags/cloud-computing.md>), [eks](<https://devfeed.tech/tags/eks.md>), [gke](<https://devfeed.tech/tags/gke.md>), [google](<https://devfeed.tech/tags/google.md>), [google-kubernetes-engine](<https://devfeed.tech/tags/google-kubernetes-engine.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [it](<https://devfeed.tech/tags/it.md>), [java](<https://devfeed.tech/tags/java.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [mainframe](<https://devfeed.tech/tags/mainframe.md>), [migration](<https://devfeed.tech/tags/migration.md>), [modernization](<https://devfeed.tech/tags/modernization.md>), [net](<https://devfeed.tech/tags/net.md>)

### AI overview

Google is consolidating its migration and modernization tools under Google Cloud Modernize, including an EKS-to-GKE Migration Agent, a Modernization Hub, and expanded capabilities for the Quick Estimator Agent. The migration agent can analyze EKS environments, translate Kubernetes manifests, map infrastructure requirements, and prepare workloads for GKE with human approval. Experts say the unified tools may simplify enterprise migration planning, while teams remain responsible for testing, architectural choices, data migration, and production cutover.

### Source excerpt

Google is overhauling its cloud modernization portfolio by bringing its existing migration and modernization tools along with new agentic migration tools and a new Modernization Hub under a new Google Cloud Modernize umbrella to accelerate infrastructure and application transformation at enterprises. The new agentic tools include an EKS-to-GKE Migration Agent, designed to help enterprises move Kubernetes workloads from Amazon EKS to Google Kubernetes Engine (GKE). The agent can analyze existing EKS environments, translate Kubernetes manifests, map storage and networking requirements, and help prepare workloads for deployment on GKE, with human approval points built into the process, the company wrote in a blog post. Alongside that agent, Google is also adding new capabilities to its Migration Center's Quick Estimator Agent. These new capabilities will allow enterprise users to generate total cost of ownership estimates from infrastructure data, including VMware inventory exports, and test different migration scenarios and cost assumptions in natural language. The Migration Center itself is being rolled into Cloud Modernize and will continue to offer tools around migration planning and assessment. Beyond the agentic capabilities and the Migration Center, the hyperscaler is consolidating its application modernization tools, such as the App Modernization CLI, Java and .NET modernization capabilities, and mainframe tools including the Mainframe Assessment Tool, Dual Run and Mainframe Connector, under a new Modernization Hub. The new Hub, which will be accessible via the Google Cloud Console, will also be a part of the umbrella offering, the company wrote. Unification of tools could simplify operations For CIOs, the consolidation of existing modernization and migration tools under a broader umbrella offering could simplify operations and decision-making as it turns what was previously a collection of separate tools into "one end-to-end process", said Pareekh Jain, princi

## How we systematically improved our reliability

DevFeed: [How we systematically improved our reliability](<https://devfeed.tech/articles/how-we-systematically-improved-our-reliability-62678.md>)

Original publisher: [Read original article](<https://neon.com/blog/how-we-systematically-improved-our-reliability>)

Author: Dmitrii Mokhnatkin

Published: 2026-09-30T12:00:00Z

Content type: article

Language: en

Sources: [Neon Docs](<https://devfeed.tech/sources/blog-neon-docs.md>)

Topics: [Production Engineering](<https://devfeed.tech/topics/production-engineering.md>), [Post Mortem](<https://devfeed.tech/topics/post-mortem.md>), [GCP outage lessons learned](<https://devfeed.tech/topics/gcp-outage-lessons-learned.md>)

Tags: [control-plane](<https://devfeed.tech/tags/control-plane.md>), [database](<https://devfeed.tech/tags/database.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [uptime](<https://devfeed.tech/tags/uptime.md>)

### AI overview

Neon describes reliability improvements to Lakebase Postgres and its control plane after two incidents caused by external dependency failures. The post outlines the operational challenge of running a large database fleet across regions and clouds, and says ongoing reliability work has improved stability as the fleet and customer base grow.

### Source excerpt

Earlier this year, we had two incidents in one week. After those incidents, we doubled down on reliability. This post summarizes the reliability work we've been rolling out since then to keep uptime high as our fleet, product, and customer base grow.

## Post-incident review for 20th October 2025

DevFeed: [Post-incident review for 20th October 2025](<https://devfeed.tech/articles/post-incident-review-for-20th-october-2025-82856.md>)

Original publisher: [Read original article](<https://buildkite.com/resources/blog/post-incident-review-for-20th-october-2025/>)

Author: The Buildkite Team

Published: 2025-10-24T19:42:00Z

Content type: article

Language: en

Sources: [Buildkite](<https://devfeed.tech/sources/buildkite.md>)

Topics: [cloud service outage management](<https://devfeed.tech/topics/cloud-service-outage-management.md>), [GCP outage lessons learned](<https://devfeed.tech/topics/gcp-outage-lessons-learned.md>), [incident management](<https://devfeed.tech/topics/incident-management.md>), [Cloud Architecture](<https://devfeed.tech/topics/cloud-architecture.md>)

Tags: [api](<https://devfeed.tech/tags/api.md>), [autoscaling](<https://devfeed.tech/tags/autoscaling.md>), [aws](<https://devfeed.tech/tags/aws.md>), [ec2](<https://devfeed.tech/tags/ec2.md>), [error](<https://devfeed.tech/tags/error.md>), [graphql](<https://devfeed.tech/tags/graphql.md>), [incident](<https://devfeed.tech/tags/incident.md>), [latency](<https://devfeed.tech/tags/latency.md>), [load-testing](<https://devfeed.tech/tags/load-testing.md>), [on-call](<https://devfeed.tech/tags/on-call.md>), [outage](<https://devfeed.tech/tags/outage.md>), [post-incident-review](<https://devfeed.tech/tags/post-incident-review.md>), [production](<https://devfeed.tech/tags/production.md>), [recovery](<https://devfeed.tech/tags/recovery.md>), [resilience](<https://devfeed.tech/tags/resilience.md>), [rest](<https://devfeed.tech/tags/rest.md>)

### AI overview

Buildkite's October 20, 2025 incident review describes how an AWS us-east-1 outage prevented the service from provisioning additional capacity as traffic rose. The resulting capacity limits caused latency and elevated errors across services, with effects varying across compute shards. Buildkite paused production deploys, communicated with customers, shifted some load to an unused shard, and prioritized capacity for the most affected shards as provisioning resumed. The company says it is investigating resilience to region-wide events and backup communication options.

### Source excerpt

How the recent AWS outage affected Buildkite services, how we responded, and what we learned.

## Lovable reports resolved GitHub integration outage and announces Labs features

DevFeed: [Lovable reports resolved GitHub integration outage and announces Labs features](<https://devfeed.tech/articles/incident-update-new-labs-features-and-more-80705.md>)

Original publisher: [Read original article](<https://lovable.dev/blog/2025-01-07-changelog>)

Author: Kristian

Published: 2025-01-07T10:00:00Z

Content type: release

Language: en

Sources: [Lovable](<https://devfeed.tech/sources/lovable.md>)

Topics: [GCP outage lessons learned](<https://devfeed.tech/topics/gcp-outage-lessons-learned.md>), [feature flags](<https://devfeed.tech/topics/feature-flags.md>), [ai-experiments](<https://devfeed.tech/topics/ai-experiments.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-powered](<https://devfeed.tech/tags/ai-powered.md>), [changelog](<https://devfeed.tech/tags/changelog.md>), [experimental](<https://devfeed.tech/tags/experimental.md>), [firecrawl](<https://devfeed.tech/tags/firecrawl.md>), [github](<https://devfeed.tech/tags/github.md>), [graphs](<https://devfeed.tech/tags/graphs.md>), [hackathon](<https://devfeed.tech/tags/hackathon.md>), [incident](<https://devfeed.tech/tags/incident.md>), [integrations](<https://devfeed.tech/tags/integrations.md>), [labs](<https://devfeed.tech/tags/labs.md>), [live-event](<https://devfeed.tech/tags/live-event.md>), [outage](<https://devfeed.tech/tags/outage.md>), [platform](<https://devfeed.tech/tags/platform.md>), [react](<https://devfeed.tech/tags/react.md>), [resolved](<https://devfeed.tech/tags/resolved.md>), [x](<https://devfeed.tech/tags/x.md>), [youtube](<https://devfeed.tech/tags/youtube.md>)

### AI overview

Lovable says a GitHub integration suspension prevented users from creating or editing projects. The issue was resolved, affected users received refunded credits, and the company announced Labs, including a Chat-Only Mode for brainstorming and troubleshooting without editing projects.

### Source excerpt

## GitHub Integration Outage On Friday, our GitHub integration was suspended, preventing users from editing or creating projects via Lovable. The issue has since been resolved, and our GitHub account is fully operational again. As a token of appreciation, we gave all users unlimited credits for the weekend. Additionally, any credits lost during the outage have been refunded. If you're still experiencing issues, please contact us via our new support page.

## How CoreDNS placement can worsen Kubernetes control plane outages

DevFeed: [How CoreDNS placement can worsen Kubernetes control plane outages](<https://devfeed.tech/articles/breaking-down-openai-s-outage-how-to-avoid-a-hidden-dns-dependency-in-kubernetes-81846.md>)

Original publisher: [Read original article](<https://render.com/blog/a-hidden-dns-dependency-in-kubernetes>)

Published: 2024-12-17T18:26:00Z

Content type: article

Language: en

Sources: [Render](<https://devfeed.tech/sources/render.md>)

Topics: [CoreDNS Kubernetes](<https://devfeed.tech/topics/coredns-kubernetes.md>), [GCP outage lessons learned](<https://devfeed.tech/topics/gcp-outage-lessons-learned.md>), [Production Engineering](<https://devfeed.tech/topics/production-engineering.md>), [Domain Name System (DNS)](<https://devfeed.tech/topics/dns-resolution-process.md>)

Tags: [control-plane](<https://devfeed.tech/tags/control-plane.md>), [dns](<https://devfeed.tech/tags/dns.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [outage](<https://devfeed.tech/tags/outage.md>), [reliability](<https://devfeed.tech/tags/reliability.md>)

### AI overview

An overloaded Kubernetes control plane can make an outage worse if DNS servers running on its nodes cannot restart. Render describes moving CoreDNS to data plane nodes and isolating etcd, changes that helped contain the impact of a later control plane failure.

### Source excerpt

Learn how to prevent Kubernetes outages by addressing hidden DNS dependencies and control plane issues. We break down OpenAI's recent incident, share key lessons from our own Kubernetes experience, and offer practical tips for improving cluster reliability and resilience.

## 2023-03-08 incident: A deep dive into the platform-level recovery

DevFeed: [2023-03-08 incident: A deep dive into the platform-level recovery](<https://devfeed.tech/articles/2023-03-08-incident-a-deep-dive-into-the-platform-level-recovery-83421.md>)

Original publisher: [Read original article](<https://www.datadoghq.com/blog/engineering/2023-03-08-deep-dive-into-platform-level-recovery/>)

Author: Laurent Bernaille

Published: 2023-06-16T00:00:00Z

Content type: article

Language: en

Sources: [Datadog](<https://devfeed.tech/sources/datadog.md>)

Topics: [Disaster Recovery](<https://devfeed.tech/topics/disaster-recovery.md>), [Service fabric](<https://devfeed.tech/topics/service-fabric.md>), [GCP outage lessons learned](<https://devfeed.tech/topics/gcp-outage-lessons-learned.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [auto-scaling](<https://devfeed.tech/tags/auto-scaling.md>), [aws](<https://devfeed.tech/tags/aws.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloudtrail](<https://devfeed.tech/tags/cloudtrail.md>), [clusters](<https://devfeed.tech/tags/clusters.md>), [cni](<https://devfeed.tech/tags/cni.md>), [container](<https://devfeed.tech/tags/container.md>), [outage](<https://devfeed.tech/tags/outage.md>)

### AI overview

The article describes Datadog's recovery of its platform after a March 8, 2023 outage disrupted compute capacity across regions. It explains the ordered recovery of Kubernetes control planes and application nodes, scaling limits in cloud networking and instance capacity, and further delays caused by AWS API rate limits, Cilium behavior, and Vault instability. The account concludes with lessons about regional isolation, capacity planning, and testing recovery at larger scales.

### Source excerpt

This post walks through how we restored our platform after it was affected by the outage of March 8, 2023.

## 2023-03-08 incident: A deep dive into the platform-level impact

DevFeed: [2023-03-08 incident: A deep dive into the platform-level impact](<https://devfeed.tech/articles/2023-03-08-incident-a-deep-dive-into-the-platform-level-impact-83420.md>)

Original publisher: [Read original article](<https://www.datadoghq.com/blog/engineering/2023-03-08-deep-dive-into-platform-level-impact/>)

Author: Laurent Bernaille

Published: 2023-05-24T00:00:00Z

Content type: article

Language: en

Sources: [Datadog](<https://devfeed.tech/sources/datadog.md>)

Topics: [GCP outage lessons learned](<https://devfeed.tech/topics/gcp-outage-lessons-learned.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>), [DevSecOps](<https://devfeed.tech/topics/devsecops.md>)

Tags: [2023](<https://devfeed.tech/tags/2023.md>), [22](<https://devfeed.tech/tags/22.md>), [auto-scaling](<https://devfeed.tech/tags/auto-scaling.md>), [availability](<https://devfeed.tech/tags/availability.md>), [aws](<https://devfeed.tech/tags/aws.md>), [azure](<https://devfeed.tech/tags/azure.md>), [ci-cd](<https://devfeed.tech/tags/ci-cd.md>), [cilium](<https://devfeed.tech/tags/cilium.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [clusters](<https://devfeed.tech/tags/clusters.md>), [incident](<https://devfeed.tech/tags/incident.md>), [outage](<https://devfeed.tech/tags/outage.md>), [platform](<https://devfeed.tech/tags/platform.md>)

### AI overview

Datadog traces a March 8, 2023 outage to a systemd-networkd behavior in Ubuntu 22.04 that flushed routing rules when a security patch restarted the service. The lost rules disrupted host connectivity across cloud providers and pod connectivity on AWS and Azure. Cloud-specific instance recovery behavior also affected data stored on local disks.

### Source excerpt

A deep dive into what happened at the platform level during the outage of March 8, 2023.

## How to Survive a Regional Outage

DevFeed: [How to Survive a Regional Outage](<https://devfeed.tech/articles/how-to-survive-a-regional-outage-28023.md>)

Original publisher: [Read original article](<https://tech.trivago.com/post/2022-06-10-how-to-survive-a-regional-outage/>)

Author: Arne Claus Follow

Published: 2022-08-15T00:00:00Z

Content type: article

Language: en

Sources: [Trivago](<https://devfeed.tech/sources/trivago.md>)

Topics: [GCP outage lessons learned](<https://devfeed.tech/topics/gcp-outage-lessons-learned.md>), [Google Cloud Platform (GCP)](<https://devfeed.tech/topics/google-cloud.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [rate-limiting](<https://devfeed.tech/topics/rate-limiting.md>), [Gke](<https://devfeed.tech/topics/gke.md>)

Tags: [2015](<https://devfeed.tech/tags/2015.md>), [2020](<https://devfeed.tech/tags/2020.md>), [2021](<https://devfeed.tech/tags/2021.md>), [akamai](<https://devfeed.tech/tags/akamai.md>), [api](<https://devfeed.tech/tags/api.md>), [asia](<https://devfeed.tech/tags/asia.md>), [backend](<https://devfeed.tech/tags/backend.md>), [backends](<https://devfeed.tech/tags/backends.md>), [booking](<https://devfeed.tech/tags/booking.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-regions](<https://devfeed.tech/tags/cloud-regions.md>), [cluster](<https://devfeed.tech/tags/cluster.md>), [code](<https://devfeed.tech/tags/code.md>), [complex](<https://devfeed.tech/tags/complex.md>), [google](<https://devfeed.tech/tags/google.md>), [google-cloud](<https://devfeed.tech/tags/google-cloud.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [outage](<https://devfeed.tech/tags/outage.md>), [region](<https://devfeed.tech/tags/region.md>)

### AI overview

trivago's global routing lets its services keep serving traffic when a Google Cloud region fails. After a backend rewrite to microservices on GKE and data movement through Kafka, the team removed most locale-based sharding and increased regional capacity. A global load balancer, tested alongside existing regional load balancers through weighted Akamai routing and a feature flag, sent requests to the nearest working region; when an Istio ingress gateway was scaled to zero, traffic shifted within seconds. The test exposed a few legacy services still dependent on regional sharding. Restoring a busy region also sent traffic back suddenly, creating a load surge that overwhelmed one advertiser. For later tests, trivago chose a less busy region and began using increase-based rate limiting, which raises its threshold over time as demand grows. The approach improved outage resilience and routing speed, though the account notes that the work depended on the earlier infrastructure migrations.

### Source excerpt

How trivago manages to stay globally available, even if a Google Cloud region goes down

## Update on 1/28 service outage

DevFeed: [Update on 1/28 service outage](<https://devfeed.tech/articles/update-on-1-28-service-outage-69743.md>)

Original publisher: [Read original article](<https://github.blog/news-insights/the-library/update-on-1-28-service-outage/>)

Author: Sam Lambert

Published: 2016-01-30T00:00:55Z

Content type: news

Language: en

Sources: [GitHub Blog](<https://devfeed.tech/sources/github-engineering.md>)

Topics: [cloud service outage management](<https://devfeed.tech/topics/cloud-service-outage-management.md>), [incident](<https://devfeed.tech/topics/incident.md>), [GCP outage lessons learned](<https://devfeed.tech/topics/gcp-outage-lessons-learned.md>)

Tags: [github](<https://devfeed.tech/tags/github.md>), [incident](<https://devfeed.tech/tags/incident.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [news-insights](<https://devfeed.tech/tags/news-insights.md>), [outage](<https://devfeed.tech/tags/outage.md>), [resilience](<https://devfeed.tech/tags/resilience.md>), [service](<https://devfeed.tech/tags/service.md>), [the-library](<https://devfeed.tech/tags/the-library.md>)

### AI overview

GitHub reports that a power disruption at its primary data center caused a cascading failure affecting several critical services. GitHub.com was unavailable for two hours and six minutes on January 28, 2016, before service was restored. The company says it is taking measures to improve resilience and response time and will share details from its investigations.

### Source excerpt

On Thursday, January 28, 2016 at 00:23am UTC, we experienced a severe service outage that impacted GitHub.com. We know that any disruption in our service can impact your development workflow,...

## GitHub outage caused by a CI configuration error that targeted production

DevFeed: [GitHub outage caused by a CI configuration error that targeted production](<https://devfeed.tech/articles/today-s-outage-70696.md>)

Original publisher: [Read original article](<https://github.blog/news-insights/the-library/today-s-outage/>)

Author: Chris Wanstrath

Published: 2010-11-15T08:30:34Z

Content type: news

Language: en

Sources: [GitHub Blog](<https://devfeed.tech/sources/github-engineering.md>)

Topics: [CI/CD](<https://devfeed.tech/topics/cicd.md>), [Continuous integration](<https://devfeed.tech/topics/continuous-integration.md>), [CI/CD secrets management](<https://devfeed.tech/topics/ci-cd-secrets-management.md>), [GCP outage lessons learned](<https://devfeed.tech/topics/gcp-outage-lessons-learned.md>), [Homelab](<https://devfeed.tech/topics/homelab.md>)

Tags: [configuration](<https://devfeed.tech/tags/configuration.md>), [continuous-integration](<https://devfeed.tech/tags/continuous-integration.md>), [data-loss](<https://devfeed.tech/tags/data-loss.md>), [database](<https://devfeed.tech/tags/database.md>), [news-insights](<https://devfeed.tech/tags/news-insights.md>), [outage](<https://devfeed.tech/tags/outage.md>), [production](<https://devfeed.tech/tags/production.md>), [testing](<https://devfeed.tech/tags/testing.md>), [the-library](<https://devfeed.tech/tags/the-library.md>)

### AI overview

A configuration error during an upgrade to GitHub's continuous integration setup caused the test environment to run against production, destroying and recreating the production database. GitHub began restoring from backup, but the large events table slowed recovery and some data might have been lost. The company planned to isolate testing from production, reduce the events table's size and growth, and store binlogs to reduce data loss during future restorations.

### Source excerpt

A few hours ago I was upgrading our continuous integration setup when a configuration error caused it to run against our production environment rather than our testing environment. Before every...

## This Morning's Git Protocol Outage

DevFeed: [This Morning's Git Protocol Outage](<https://devfeed.tech/articles/this-morning-s-git-protocol-outage-70712.md>)

Original publisher: [Read original article](<https://github.blog/news-insights/the-library/this-morning-s-git-protocol-outage/>)

Author: Tom Preston-Werner

Published: 2010-09-29T00:48:16Z

Content type: news

Language: en

Sources: [GitHub Blog](<https://devfeed.tech/sources/github-engineering.md>)

Topics: [GCP outage lessons learned](<https://devfeed.tech/topics/gcp-outage-lessons-learned.md>), [cloud service outage management](<https://devfeed.tech/topics/cloud-service-outage-management.md>), [infrastructure monitoring](<https://devfeed.tech/topics/infrastructure-monitoring.md>)

Tags: [availability](<https://devfeed.tech/tags/availability.md>), [git](<https://devfeed.tech/tags/git.md>), [haproxy](<https://devfeed.tech/tags/haproxy.md>), [http](<https://devfeed.tech/tags/http.md>), [news-insights](<https://devfeed.tech/tags/news-insights.md>), [outage](<https://devfeed.tech/tags/outage.md>), [the-library](<https://devfeed.tech/tags/the-library.md>), [timeout](<https://devfeed.tech/tags/timeout.md>)

### AI overview

Anonymous Git clones were unavailable for several hours after a change removed HAProxy's client inactivity timeout, allowing idle connections to exhaust the session limit. SSH and HTTP clones remained available. The timeout was restored, and GitHub said it was improving monitoring and reviewing future changes.

### Source excerpt

Anonymous Git clones were offline this morning for several hours (SSH and HTTP clones remained functional). Several days ago we removed the client inactivity timeout from our backend HAProxy configuration...