# recovery Published articles for recovery. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## How to operate shared platforms safely at agent scale DevFeed: [How to operate shared platforms safely at agent scale]() Original publisher: [Read original article]() Author: Candace Shamieh; T Zhang; Gabriele Baldoni Published: 2026-09-15T00:00:00Z Content type: article Language: en Sources: [Datadog | The Monitor blog]() Topics: [Platform Engineering](), [Loop Engineering](), [AI Agent](), [sdlc](), [Latency](), [Low Latency](), [Concurrency](), [Software]() Tags: [agent](), [agent-observability](), [agentic-workflows](), [agents](), [ai](), [ai-agent](), [ci](), [concurrency](), [latency](), [low-latency](), [operational](), [performance](), [platform-engineering](), [platforms](), [queue](), [recovery](), [timeout]() ### AI overview This Datadog article explains how platform teams can operate shared platforms safely as AI agent workloads scale across teams. It discusses modeling demand across agent trajectories, planning capacity across dependencies such as CI queues and sandbox pools, handling contention and recovery behavior, and preserving control across system boundaries. ### Source excerpt Learn how Datadog models agent demand, allocates capacity under contention, and preserves control as AI agent workloads scale across shared platforms. ## What Managed Postgres Services Handle for Teams DevFeed: [What Managed Postgres Services Handle for Teams]() Original publisher: [Read original article]() Author: Databricks Staff Published: 2026-09-14T23:36:59Z Content type: article Language: en Sources: [Databricks]() Topics: [PostgreSQL](), [Database](), [Serverless](), [databricks](), [AI Development]() Tags: [data-plus-ai-foundations](), [database](), [databricks](), [developer-tooling](), [migration](), [postgres](), [postgresql](), [recovery](), [security](), [serverless]() ### AI overview The article defines managed Postgres by the operational responsibilities a provider assumes, including patching, scaling, failover, and backups. It discusses how these responsibilities can vary between providers and describes Lakebase Postgres as a serverless offering with automatic scaling, PostgreSQL compatibility, recovery, and Databricks integrations. ### Source excerpt Every Postgres vendor calls itself "managed." Few of them agree on what that word... ## How We Built Automated Capacity Testing for Kafka Consumers DevFeed: [How We Built Automated Capacity Testing for Kafka Consumers]() Original publisher: [Read original article]() Author: Kaan Karakaya Published: 2026-09-14T09:46:34Z Content type: tutorial Language: en Sources: [Booking.com Development - Medium]() Topics: [Kafka](), [Testing](), [HTTP]() Tags: [automated](), [capacity](), [health-checks](), [kafka](), [load](), [load-balancer](), [parallelism](), [partition](), [recovery](), [scale](), [site-reliability-engineer](), [sre](), [testing]() ### AI overview This article describes Booking.com's extension of an existing capacity-testing platform for Kafka consumers. It explains how changing partition assignment can provide a controlled, measurable way to test consumer throughput and whether remaining consumers can absorb reassigned work after an instance or failure domain disappears. ### Source excerpt Photo by GuerrillaBuzz on Unsplash Kafka makes it easy to distribute work across consumer instances. It is much harder to prove, safely and repeatedly, how those instances behave when the distribution changes and one of them has to carry more than its usual share. For teams that run Kafka at scale, this is a practical reliability question: how much load can a consumer instance actually handle? We had automated capacity testing for HTTP services, but Kafka consumers were still tested with manual drills. Those drills could tell us something, but they were disruptive, difficult to reproduce, and risky precisely when the system was close to its limit. We wanted a controlled way to answer three questions: What is the maximum sustainable throughput of a consumer instance? If an instance or failure domain disappears, can the remaining consumers absorb the reassigned work? Are we overprovisioning resources because we do not know the real limit? The result was an extension to our capacity-testing platform that turns Kafka partition assignment into a safe, measurable load-control mechanism. Why HTTP capacity testing did not translate Our existing platform was designed for request-response services behind a load balancer. A scheduled test selects one instance, routes an increasing share of traffic to it, runs health checks after each step, and records the highest ratio the instance can sustain. After the test, traffic returns to its normal distribution and the result is reported to the service owner. Kafka has no equivalent traffic knob. Consumers pull records, and the unit of parallelism is the partition. Within a consumer group, each partition is owned by one consumer at a time. If a topic has 12 partitions and four equally loaded instances, each instance owns about three. When one instance disappears, a rebalance gives the survivors more partitions -- and the extra work arrives as a step change, not as a smooth increase from a load balancer. The key translation: for an HTTP ## BTRFS boot failure and easy GUI methods for system recovery DevFeed: [BTRFS boot failure and easy GUI methods for system recovery]() Original publisher: [Read original article]() Author: Alex Genovese Published: 2026-09-04T20:54:40Z Content type: article Language: en Sources: [Fedora Magazine]() Topics: [boot](), [Fedora](), [GUI](), [Linux](), [dnf](), [Software]() Tags: [backup](), [boot](), [btrfs](), [desktop](), [dnf](), [faqs-and-guides](), [fedora-project-community](), [for-system-administrators](), [gui](), [how-to](), [installation](), [iso](), [linux](), [recovery](), [snapshots](), [usb](), [using-hardware](), [using-software](), [workstation]() ### AI overview This how-to article explains GUI-based recovery options for Fedora Linux systems using BTRFS. It covers booting from a USB live ISO, checking whether the operating system uses BTRFS, and recovering earlier system states with snapshot tools such as btrfs-assistant and snapper. ### Source excerpt A few short how-to's with screenshots, showing easy ways to boot up off a USB live ISO and return back to previous points in time with BTRFS ## Omdia Study Highlights Declining Ransomware Recovery Rates and the Immutability Gap in Backup Storage DevFeed: [Omdia Study Highlights Declining Ransomware Recovery Rates and the Immutability Gap in Backup Storage]() Original publisher: [Read original article]() Author: Harold Fritts Published: 2026-09-01T15:42:11Z Content type: news Language: en Sources: [StorageReview.com]() Topics: [ransomware](), [Resilience](), [Security]() Tags: [backup](), [data-protection](), [enterprise](), [ransomware](), [recovery](), [research](), [resilience](), [security](), [third-party]() ### AI overview An Omdia study commissioned by Object First reports declining enterprise ransomware recovery rates despite increased awareness of data protection strategies. The findings show frequent attacks, worsening data recoverability, missed recovery objectives, and a substantial gap between the perceived importance of immutable backup storage and its actual implementation. The study also highlights demand for independent validation of vendor immutability claims. ### Source excerpt A recent study by the analyst firm Omdia, commissioned by Object First, reveals that enterprise ransomware recovery rates are declining despite growing awareness of modern data protection strategies. According to the research, 83 percent of surveyed organizations experienced a successful ransomware attack in the past 24 months, up from 66 percent in 2024. Among those The post Omdia Study Highlights Declining Ransomware Recovery Rates and the Immutability Gap in Backup Storage appeared first on StorageReview.com. ## Data Engineering Weekly #285 DevFeed: [Data Engineering Weekly #285]() Original publisher: [Read original article]() Author: Ananth Packkildurai Published: 2026-08-31T02:51:19Z Content type: article Language: en Sources: [Data Engineering Weekly]() Topics: [data-engineering](), [AI, ML & Data Engineering](), [Data Quality](), [observability](), [Artificial Intelligence]() Tags: [agentic-ai](), [automated](), [data-engineering](), [newsletter](), [observability](), [privacy](), [python](), [quality](), [recovery](), [weekly]() ### AI overview Data Engineering Weekly #285 covers building data platforms, AI chip architectures, preparing data for agentic AI, post-AI data stacks, data modernization, automated data contract breach handling, and privacy-preserving measurement tools. ### Source excerpt The Weekly Data Engineering Newsletter ## Paddle vs Lemon Squeezy: Same 5% + 50c, Very Different Bets DevFeed: [Paddle vs Lemon Squeezy: Same 5% + 50c, Very Different Bets]() Original publisher: [Read original article]() Author: Deepak Jangir Published: 2026-08-28T00:00:00Z Content type: comparison Language: en Sources: [Dodo Payments Blog]() Topics: [stripe](), [Software as a service]() Tags: [2026](), [comparison](), [digital-products](), [fraud](), [merchant-of-record](), [payments](), [pricing](), [recovery](), [saas](), [support](), [tax]() ### AI overview This comparison argues that Paddle and Lemon Squeezy have the same headline Merchant of Record pricing but target different customers. Paddle is positioned for B2B SaaS at scale, while Lemon Squeezy is aimed at indie makers and digital products. The article also explains how Stripe's acquisition of Lemon Squeezy and the rollout of Stripe Managed Payments change the comparison in 2026. ### Source excerpt Paddle and Lemon Squeezy both charge 5% + 50c as Merchant of Record. Compare who they serve, what changed after Stripe bought Lemon Squeezy, and the real costs. ## 【etcd】运维与升级：member change、backup/restore 与 3.5-\>3.6/3.7 门 DevFeed: [【etcd】运维与升级：member change、backup/restore 与 3.5-\>3.6/3.7 门]() Original publisher: [Read original article]() Author: Liao Tonglang Published: 2026-08-26T00:00:00Z Content type: tutorial Language: zh Sources: [土法炼钢 - 系统与基础设施]() Topics: [etcd](), [upgrade](), [Kubernetes](), [migration]() Tags: [backup](), [distributed](), [etcd](), [kubernetes](), [learner](), [member-change](), [migration](), [operations](), [raft](), [recovery](), [restore](), [snapshot](), [upgrade](), [v3](), [v3-5](), [v3-5-33](), [v3-6](), [v3-7]() ### AI overview A Chinese-language operational guide to etcd v3.5.33 covering member replacement and learner promotion, live snapshot backup and restore, Kubernetes control-plane recovery, patch upgrades, and documented upgrade gates from etcd 3.5 to 3.6 and 3.7. ### Source excerpt member add/remove/replace 与 learner promote 检查单；snapshot backup/restore 及 K8s 语境 revision bump；3.5.33 补丁线与 3.6/3.7 官方升级门禁项，不超前写未钉 tag 能力。 ## 【etcd】treeIndex 与 Apply 管道：propose -\> commit -\> apply DevFeed: [【etcd】treeIndex 与 Apply 管道：propose -\> commit -\> apply]() Original publisher: [Read original article]() Author: Liao Tonglang Published: 2026-08-26T00:00:00Z Content type: tutorial Language: zh Sources: [土法炼钢 - 系统与基础设施]() Topics: [etcd](), [Raft](), [Back end]() Tags: [applied-index](), [apply](), [backend](), [bbolt](), [commit](), [compaction](), [distributed](), [etcd](), [index](), [kubernetes](), [mvcc](), [raft](), [range](), [recovery](), [revision](), [store](), [treeindex](), [txn](), [v3-5](), [wal](), [watch]() ### AI overview This tutorial explains the etcd v3.5.33 apply pipeline from Raft commit to MVCC application. It covers the separation between propose, commit, and apply; the roles of treeIndex and bbolt; consistent-index updates; watch notification timing; and troubleshooting committed-versus-applied lag. ### Source excerpt 走读 etcd v3.5.33 从 Raft commit 到 MVCC apply 的串行管道：treeIndex 与 bbolt 分工、consistent index、watchableStore 通知触发点，以及 committed/applied 分列排障。 ## Configuring compaction thresholds and context windows for coding agents DevFeed: [Configuring compaction thresholds and context windows for coding agents]() Original publisher: [Read original article]() Author: WorkOS Published: 2026-08-14T19:33:33Z Content type: tutorial Language: en Sources: [WorkOS Blog]() Topics: [coding](), [context window](), [long-context]() Tags: [agent](), [coding](), [context-window](), [generation](), [model](), [recovery](), [tokens]() ### AI overview The article explains how to derive a coding agent's effective context window from compaction thresholds and the response runway required to complete generation. It discusses threshold behavior, model metadata, route limits, generation clamping, overflow detection, and recovery. ### Source excerpt Derive your coding agent's effective context window from two numbers: the compaction threshold you want, and the response runway the model needs to finish. ## How MongoDB Atlas and Temporal support reliable production RAG and AI agents DevFeed: [How MongoDB Atlas and Temporal support reliable production RAG and AI agents]() Original publisher: [Read original article]() Author: Suresh Ramappa Published: 2026-08-13T00:00:00Z Content type: opinion Language: en Sources: [Temporal Blog]() Topics: [MongoDB](), [Retrieval Augmented Generation (RAG)](), [AI Infrastructure](), [Artificial Intelligence](), [reliability]() Tags: [agentic](), [agents](), [ai](), [ai-infrastructure](), [api](), [data](), [embeddings](), [long-running](), [mongodb](), [outages](), [pipelines](), [production](), [rag](), [recovery](), [reliability](), [retries](), [retrieval](), [temporal-voices]() ### AI overview The article argues that MongoDB Atlas and Temporal address different reliability needs in production RAG and AI agent systems. Atlas provides operational data, embeddings, vector search, and agent memory in one platform, while Temporal provides durable execution for crash recovery, retries, and long-running ingestion and agent workflows. ### Source excerpt Why MongoDB Atlas and Temporal are better together for AI: one data platform, one durable execution layer, for RAG and agents in prod. ## Multi-Region PostgreSQL Disaster Recovery and Failback with Crunchy PGO DevFeed: [Multi-Region PostgreSQL Disaster Recovery and Failback with Crunchy PGO]() Original publisher: [Read original article]() Author: Wellingtone Luvonga Published: 2026-08-11T05:00:00Z Content type: tutorial Language: en Sources: [CYBERTEC PostgreSQL | Services & Support]() Topics: [PostgreSQL](), [Disaster Recovery](), [Kubernetes](), [MinIO](), [nginx](), [TLS (Transport Layer Security)](), [Amazon S3](), [Availability]() Tags: [availability](), [backup](), [data-recovery](), [disaster-recovery](), [failover](), [high-availability](), [how-to](), [kubernetes](), [minio](), [nginx](), [postgresql](), [recovery](), [s3](), [tls](), [tutorials]() ### AI overview A tutorial on using Crunchy PostgreSQL Operator on Kubernetes to implement multi-region PostgreSQL disaster recovery and failback. It covers secure MinIO-backed WAL storage through an NGINX TLS reverse proxy, regional failover, and reversing roles without timeline conflicts or archive poisoning. ### Source excerpt This blog talks about multiregion PostgreSQL disaster recovery using Crunchy PGO. It has some detailed steps to understand better. The post Multi-Region PostgreSQL Disaster Recovery and Failback with Crunchy PGO appeared first on CYBERTEC PostgreSQL | Services & Support. ## Passkeys in B2B: the real risk is recovery DevFeed: [Passkeys in B2B: the real risk is recovery]() Original publisher: [Read original article]() Author: WorkOS Published: 2026-08-06T15:29:33Z Content type: article Language: en Sources: [WorkOS Blog]() Topics: [Passkeys](), [WebAuthn](), [Cloud](), [systems]() Tags: [b2b](), [cloud](), [cryptographic](), [phishing](), [recovery]() ### AI overview The article explains why consumer passkey guidance does not fully apply to B2B systems. Although passkeys are phishing-resistant WebAuthn credentials, synced passkeys may reside in employees' personal cloud accounts, creating enrollment, ownership, recovery, and revocation challenges when devices are lost or employees leave. ### Source excerpt Consumer passkey guidance optimizes for the login. In B2B, the enrollment and recovery story is what decides whether passkeys actually work. Here's the gap. ## Disaster Recovery Testing Best Practices for 2026 DevFeed: [Disaster Recovery Testing Best Practices for 2026]() Original publisher: [Read original article]() Author: Pritesh Kiri Published: 2026-08-04T00:00:00Z Content type: article Language: en Sources: [Harness Blog]() Topics: [Disaster Recovery](), [Testing](), [Resilience](), [Automation]() Tags: [2026](), [audit](), [automated](), [automation](), [best-practices](), [cross-functional-teams](), [disaster-recovery](), [external](), [metrics](), [pipelines](), [recovery](), [resilience](), [testing]() ### AI overview This article explains how to evolve disaster recovery testing from reactive exercises into a mature, metrics-driven resilience program. It covers risk-aligned testing schedules, automation, continuous improvement, and metrics for demonstrating recovery effectiveness. ### Source excerpt Learn how to move from ad hoc DR tests to a mature, metrics-driven resilience program with risk-aligned schedules, automation, and the KPIs that prove your reco | Blog ## Safe Database Rollback Starts Before Deployment DevFeed: [Safe Database Rollback Starts Before Deployment]() Original publisher: [Read original article]() Author: Raul Junco Published: 2026-07-25T12:00:54Z Content type: tutorial Language: en Sources: [System Design Classroom]() Topics: [Database](), [Deployment](), [migration](), [feature flags]() Tags: [database](), [deployment](), [feature-flags](), [migrations](), [production](), [recovery](), [rollback](), [schema]() ### AI overview This article explains why rolling back application code does not automatically roll back database schema or production data. It presents backward-compatible schema changes, staged migrations, feature flags, and targeted recovery as ways to prepare for deployment failures and reduce the risk of data loss. ### Source excerpt Learn why safe database rollback starts before deployment, and how backward-compatible schema changes, staged migrations, feature flags, and targeted recovery keep production failures from turning into data loss. ## Postgres backups under the hood DevFeed: [Postgres backups under the hood]() Original publisher: [Read original article]() Author: Josh Brown Published: 2026-07-24T00:00:00Z Content type: article Language: en Sources: [Blog -- PlanetScale]() Topics: [Databases](), [data](), [SQL](), [IO](), [Filesystems]() Tags: [backup](), [data](), [database](), [engineering](), [how-to](), [io](), [postgres](), [recovery](), [sql]() ### AI overview This article explains how Postgres backups work, focusing on logical backups and their tradeoffs. It describes consistent snapshots, SQL or custom output formats, resource costs at large data volumes, the lack of point-in-time recovery, and the use of MVCC for online backups. ### Source excerpt With backups being such a vital part of keeping your data safe, how do they actually work? ## Using AI to Troubleshoot Kubernetes Clusters DevFeed: [Using AI to Troubleshoot Kubernetes Clusters]() Original publisher: [Read original article]() Author: Nimesha Jinarajadasa Published: 2026-07-18T08:34:19Z Content type: tutorial Language: en Sources: [Kubernetes - KodeKloud Blog | DevOps, Cloud, Kubernetes, AI Tutorials & More]() Topics: [Kubernetes](), [Artificial Intelligence]() Tags: [ai](), [clusters](), [errors](), [guide](), [how-to](), [kubernetes](), [recovery](), [troubleshooting](), [verify]() ### AI overview A practical tutorial on using k8sgpt to troubleshoot deliberately broken Kubernetes clusters. It covers diagnosing ImagePullBackOff and CrashLoopBackOff errors, generating AI-assisted explanations and fix steps, applying a suggested fix, and verifying recovery while a human remains responsible for the decision. ### Source excerpt Use AI to troubleshoot Kubernetes with k8sgpt: diagnose real ImagePullBackOff and CrashLoopBackOff errors, get fix steps, and verify recovery. ## Recover the Ceph Monitor Store Using OSDs DevFeed: [Recover the Ceph Monitor Store Using OSDs]() Original publisher: [Read original article]() Author: Eugen Block, crossposted by Anthony D'Atri Published: 2026-07-04T00:00:00Z Content type: article Language: en Sources: [Ceph Blog]() Topics: [ceph](), [monitor](), [Script](), [Containers](), [log management]() Tags: [blog](), [blog-post](), [ceph](), [containers](), [en-article](), [en-blog-post](), [how-to](), [logging](), [logs](), [mon](), [monitor](), [recovery]() ### AI overview This blog post presents a general procedure for recovering a failed Ceph Monitor store by collecting osdmaps from existing OSDs, with additional considerations for cephadm-managed container deployments. It automates osdmap collection and basic logging while leaving the store rebuild and client authentication steps for careful manual inspection. The procedure applies only to non-encrypted OSDs, and the article emphasizes the importance of backing up dmcrypt/LUKS keys and reviewing host logs. ### Source excerpt Introduction ¶ A few weeks ago I helped a Ceph user to recover his broken cluster (see this thread). Basically, after his Monitors stopped working he re-deployed a new cluster with the same Ceph FSID and attached the existing OSDs to the re-deployed hosts. But it's not that easy to re-activate those OSDs because the new Monitors don't have the old osdmap, hence they don't know anything about the existing OSDs. So how did we fix that? This scenario can be considered a total Monitor store failure. There's a documented procedure in the upstream docs, unfortunately it's written for non-cephadm clusters and doesn't contain many details about subsequent steps. When Ceph daemons run within containers there are more things to consider. So I decided to write this blog post and add some more details to the procedure, targeting mainly clusters managed by cephadm. But the recovery procedure doesn't specifically require cephadm-specific commands, so it can be considered as a general guideline how to recover from a Monitor store loss, just with some extra details about cephadm deployments. This procedure only works for non-encrypted OSDs. If you are using dmcrypt (LUKS) OSDs, make sure you have a backup of the keys, otherwise your data is lost forever! But there is some development, a user in Slack pointed me to this PR which adds a backup mechanism for Monitors, probably available in the upcoming Umbrella release. Note that the PR also contains this statement: Monitor backups complement, but do not replace, the existing Monitor recovery procedures The procedure covered by said docs is already written in script form, so I used that as a template and extended it for cephadm usage and included some very basic logging. It collects the osdmaps from all OSDs and contains the necessary considerations regarding containers. I decided to automate only the osdmap collection, not all required steps of the store rebuild procedure (e. g. Monitor store rebuild, client auth, etc.) because the r ## Subscription Management Software: A 2026 Buyer's Guide DevFeed: [Subscription Management Software: A 2026 Buyer's Guide]() Original publisher: [Read original article]() Author: Aarthi Poonia Published: 2026-06-28T00:00:00Z Content type: article Language: en Sources: [Dodo Payments Blog]() Topics: [Software](), [Software as a service](), [Finance]() Tags: [2026](), [billing](), [churn](), [compare](), [compliance](), [features](), [guide](), [lifecycle](), [payment](), [pricing](), [recovery](), [reporting](), [retention](), [revenue](), [saas](), [subscription-billing](), [subscriptions](), [tax]() ### AI overview A 2026 buyer's guide to subscription management software, covering its role across the recurring-customer lifecycle, must-have billing and operational features, leading options, and platform selection for SaaS businesses. ### Source excerpt What subscription management software does, the must-have features, top tools compared, and how to choose the right platform for your SaaS in 2026. ## The Matter upgrade you've been waiting for DevFeed: [The Matter upgrade you've been waiting for]() Original publisher: [Read original article]() Author: Ingo Fischer Published: 2026-06-23T00:00:00Z Content type: article Language: en Sources: [Home Assistant]() Topics: [Matter](), [Home Assistant](), [TypeScript](), [Open Source](), [Network](), [Homebridge]() Tags: [2026](), [announcements](), [border-router](), [bugs](), [hardware](), [homebridge](), [matter](), [open-source](), [recovery](), [speed](), [standards](), [thread](), [typescript](), [upgrade](), [visualization]() ### AI overview Home Assistant introduces a new Matter implementation built on matter.js, an open source TypeScript implementation of the Matter standard. After beta testing and community contributions, the update aims to improve stability, reduce bugs, and accelerate startup and recovery. It also adds network visualization, upgrades to Matter 1.5.1, and updates the OpenThread Border Router app for Thread 1.4 support. ### Source excerpt It started as a passion project. Yet in a few years matter.js has grown beyond my wildest expectations, becoming the backbone of the open source Matter ecosystem, and powering everything from Homebridge to openHAB, and even some commercial products. Ten months ago, I joined forces with the Open Home Foundation to work full time as Lead Developer of Matter. Shortly after, I donated matter.js to the foundation - giving it a new home where it would be safeguarded for the future. Together we announced our plans to take Matter support in Home Assistant in a completely new direction: we set out to rebuild the Open Home Foundation Matter Server - the engine that powers Matter in Home Assistant - on matter.js. After four months of beta testing, with our community fixing bugs and contributing new features along the way, today we're thrilled to present the newest version of Matter for Home Assistant! 🎉 The Open Home Foundation invests in important technology to make open alternatives possible, and our work is funded in part by everyone who subscribes to Home Assistant Cloud and buys official hardware from our commercial partners. The launch of a new Matter experience in Home Assistant shows what's possible with your support. 💪 Presenting the matter.js plans at the Connectivity Standards Alliance (CSA) Member Meeting in November 2025 (right), and returning in March 2026 as an Outstanding Contributor Award recipient (left). Serving up more with matter.js For those new to matter.js, it's an open source TypeScript implementation of the Matter standard. I built it to give developers and power users more speed and flexibility, and to make Matter better suited for open source development across the millions of homes that use Home Assistant. By running Matter in Home Assistant on matter.js, all of those benefits now flow directly into the platform. This major update brings greater stability for Matter in Home Assistant, fewer bugs, and faster start-up and recovery. It also comes with ## Meet Alice. Alice is impatient. DevFeed: [Meet Alice. Alice is impatient.]() Original publisher: [Read original article]() Author: Marc Brooker Published: 2026-06-19T00:00:00Z Content type: article Language: en Sources: [Marc Brooker's Blog](), [Marc Brooker's Blog]() Topics: [Latency](), [Web](), [Code](), [Server]() Tags: [code](), [latency](), [metrics](), [outage](), [reasoning](), [recovery](), [server](), [time](), [web]() ### AI overview The article explains why customers may perceive web-service latency and outages as much longer than service-side averages suggest. It uses the inspection paradox to show that people disproportionately encounter longer events, presents a formula for the customer-observed mean recovery time, and describes a simulator based on median and p99 values. ### Source excerpt Meet Alice. Alice is impatient. What do you mean? Meet Alice. Alice uses your web service. Alice, like most humans, measures her time in seconds and minutes. Alice says your service is slow. You tell Alice that the mean request to your service completes in 100ms, but Alice says that her mean wait time is 1s. You're both right. Meet Alex. Alex uses your web service. Alex, like most humans, measures his time in seconds and minutes. Alex says that when you have outages, they last a long time and he gets really annoyed. You tell Alex that your MTTR is less than 1 minute. Alex says that he sees the mean outage lasting 1 hour. Again, you're both right. What's going on? What's going on is that you're measuring time in requests, or in outages, and Alex and Alice are measuring time in seconds and minutes. When you have a long pause or a long outage, Alex and Alice sample that outage multiple times (maybe because they have multiple customers angry at them). The number of times they experience the outage is proportional to the length of the outage. But you only count that as one. More technically, what's going on here is the inspection paradox. Alex and Alice don't experience your latency distribution $f(t)$, they experience a t-weighted version of it. If you have a MTTR or mean request time of $\mathbb{E}[X]$, Alex and Alice experience a mean recovery time $\mathbb{E}_a[X]$ where $\mathbb{E}_a[X] = \frac{\mathbb{E}[X^2]}{2 \mathbb{E}[X]} = \frac{1}{2} \left( \mathbb{E}[X] + \frac{\mathrm{Var}(X)}{\mathbb{E}[X]} \right)$. Let's play with this with a little simulation. Plug in your median latency (or recovery time), and 99th percentile latency (or recovery time), we'll fit a log-normal distribution to it, and then plot both what your service metrics see and what your customers see. Median: ms p99: ms What your service sees (mean): - ms. What your customers experience (mean): - ms. For example, put in 30 as the median (let's ignore the milliseconds and pretend these are minutes ## Coordinate access to shared resources with a distributed lock built on Temporal Workflows DevFeed: [Coordinate access to shared resources with a distributed lock built on Temporal Workflows]() Original publisher: [Read original article]() Author: Keith Tenzer Published: 2026-06-11T00:00:00Z Content type: tutorial Language: en Sources: [Temporal Blog]() Topics: [locking]() Tags: [distributed](), [lease](), [locking](), [locks](), [recovery](), [resource](), [temporal-concepts](), [workflows]() ### AI overview This guide describes a reusable, durable distributed lock for Temporal Workflows. It models each permit as a short-lived child Workflow, uses Workflow IDs for atomic acquisition, and uses Signals and lease timeouts to release locks and recover orphaned permits. ### Source excerpt This guide details a reusable, durable distributed lock for Temporal Workflows that doesn't rely on an external database, central limiter, or any shared state. ## Five rules for running an incident DevFeed: [Five rules for running an incident]() Original publisher: [Read original article]() Author: Sridhar Rajarao Published: 2026-05-27T00:00:00Z Content type: opinion Language: en Sources: [Sridhar Rajarao]() Topics: [incident](), [Learning]() Tags: [customer-experience](), [incident](), [incident-management](), [on-call](), [outage](), [postmortem](), [production](), [recovery](), [reliability](), [root-cause-analysis](), [rules](), [signal](), [speed](), [sre](), [team]() ### AI overview This opinion article presents five rules for handling production incidents: assess severity by customer impact, use an Incident Commander, mitigate before investigating root cause, maintain regular communication, and use postmortems for learning and accountable follow-up. ### Source excerpt The difference between a 10-minute incident and a 3-hour outage is rarely technical. Five things I wish every on-call team locked in before their first big page. ## Tracking Down Orphaned Objects in ClickHouse Clusters and Improving Cleanup and Recovery DevFeed: [Tracking Down Orphaned Objects in ClickHouse Clusters and Improving Cleanup and Recovery]() Original publisher: [Read original article]() Author: Irene Martínez Published: 2026-05-19T00:00:00Z Content type: article Language: en Sources: [Tinybird]() Topics: [clickhouse](), [data](), [incident](), [cloud-infrastructure]() Tags: [cleanup](), [clickhouse](), [clusters](), [data](), [engineering-excellence](), [incident](), [recovery](), [storage]() ### AI overview This article describes how the team tracked down orphaned objects in its ClickHouse clusters, reduced monthly storage costs, and improved cleanup and recovery procedures after a near data-loss incident. ### Source excerpt How we tracked down petabytes of orphaned objects in our ClickHouse clusters, recovered tens of thousands of dollars in monthly storage costs, and hardened our cleanup and recovery procedures after almost losing data along the way. [Next page]()