# site-reliability-engineer Published articles for site-reliability-engineer. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## How end-to-end SLO monitoring detected a livestream failure that component dashboards missed DevFeed: [How end-to-end SLO monitoring detected a livestream failure that component dashboards missed]() Original publisher: [Read original article]() Author: Whatnot Engineering Published: 2026-09-15T16:31:01Z Content type: article Language: en Sources: [Whatnot Engineering]() Topics: [dashboards](), [observability](), [incident](), [Monitoring](), [Web](), [client](), [Network]() Tags: [dashboards](), [distributed-systems](), [incident](), [monitoring](), [observability](), [security](), [server](), [site-reliability-engineer](), [software-engineering](), [web]() ### AI overview The article examines a June 8, 2026 incident in which a third-party client-side security script fetched from a provider CDN added URL validation that the video provider did not pass, causing black screens for newly loaded web clients. Most component-level dashboards remained green, while end-to-end service-level objective monitoring detected the broken livestream experience and paged the owning teams within five minutes. ### Source excerpt Karol Gil | Reliability Platform (Poland) On June 8, 2026, newly loaded web clients began showing black screens instead of livestream video. For our platform, that's a serious problem: it's pretty hard to sell Pokémon cards that no one can see. It turned out that a third-party script we use for client-side security monitoring wasn't bundled with our release, but was rather fetched live from the provider's CDN. When the provider updated the script all new web clients fetched it, and it included an additional URL validation which our video provider didn't pass. The result? Black screens for users of the affected web clients, with most internal dashboards staying green. 3,000 users were impacted in the first 30 minutes of the incident. One system did catch it. Our end-to-end service-level objective (E2E SLO) monitoring was already in production and paged the owning teams within five minutes. Here's what it saw. The real problem Most of our dashboards stayed green because they monitor component-level health: a server, an endpoint, a specific function. These are all useful, but can all be healthy while the actual user experience is completely broken. This problem gets worse the more external dependencies there are, or the more sophisticated an experience you want to deliver. In complex, integrated product experiences like ours, a "small" problem can have an outsize impact on the user experience. Measuring this requires a different approach to observability, namely, to model the user journey across multiple surfaces that must be true for a customer to have a good experience. So how do we measure this in a complex distributed application? Joining a livestream is not one thing Joining a livestream sounds like one action, but the user expects at least three things: Video to be playing Auction details to be shown Chat to be visible and up to da Each of those can succeed or fail completely independently of the other two. Our video depends on third-party providers and CDN netwo ## How We Built Automated Capacity Testing for Kafka Consumers DevFeed: [How We Built Automated Capacity Testing for Kafka Consumers]() Original publisher: [Read original article]() Author: Kaan Karakaya Published: 2026-09-14T09:46:34Z Content type: tutorial Language: en Sources: [Booking.com Development - Medium]() Topics: [Kafka](), [Testing](), [HTTP]() Tags: [automated](), [capacity](), [health-checks](), [kafka](), [load](), [load-balancer](), [parallelism](), [partition](), [recovery](), [scale](), [site-reliability-engineer](), [sre](), [testing]() ### AI overview This article describes Booking.com's extension of an existing capacity-testing platform for Kafka consumers. It explains how changing partition assignment can provide a controlled, measurable way to test consumer throughput and whether remaining consumers can absorb reassigned work after an instance or failure domain disappears. ### Source excerpt Photo by GuerrillaBuzz on Unsplash Kafka makes it easy to distribute work across consumer instances. It is much harder to prove, safely and repeatedly, how those instances behave when the distribution changes and one of them has to carry more than its usual share. For teams that run Kafka at scale, this is a practical reliability question: how much load can a consumer instance actually handle? We had automated capacity testing for HTTP services, but Kafka consumers were still tested with manual drills. Those drills could tell us something, but they were disruptive, difficult to reproduce, and risky precisely when the system was close to its limit. We wanted a controlled way to answer three questions: What is the maximum sustainable throughput of a consumer instance? If an instance or failure domain disappears, can the remaining consumers absorb the reassigned work? Are we overprovisioning resources because we do not know the real limit? The result was an extension to our capacity-testing platform that turns Kafka partition assignment into a safe, measurable load-control mechanism. Why HTTP capacity testing did not translate Our existing platform was designed for request-response services behind a load balancer. A scheduled test selects one instance, routes an increasing share of traffic to it, runs health checks after each step, and records the highest ratio the instance can sustain. After the test, traffic returns to its normal distribution and the result is reported to the service owner. Kafka has no equivalent traffic knob. Consumers pull records, and the unit of parallelism is the partition. Within a consumer group, each partition is owned by one consumer at a time. If a topic has 12 partitions and four equally loaded instances, each instance owns about three. When one instance disappears, a rebalance gives the survivors more partitions -- and the extra work arrives as a step change, not as a smooth increase from a load balancer. The key translation: for an HTTP ## Expedia's Service Telemetry Analyzer DevFeed: [Expedia's Service Telemetry Analyzer]() Original publisher: [Read original article]() Author: Nikos Katirtzis Published: 2026-04-28T11:01:01Z Content type: article Language: en Sources: [Expedia]() Topics: [telemetry](), [distributed-systems](), [Prompt Engineering](), [Artificial Intelligence](), [API](), [Cloud](), [Containers](), [Orchestration]() Tags: [ai](), [api](), [distributed-systems](), [generative-ai-tools](), [metric](), [observability](), [prompt-engineering](), [root-cause-analysis](), [site-reliability-engineer](), [software-engineering](), [telemetry]() ### AI overview Expedia's Service Telemetry Analyzer (STAR) is an early web-based system for investigating service degradations and outages with service telemetry data and AI models. It uses predefined multi-step diagnostic workflows, domain-specific prompt engineering, and engineering knowledge spanning applications, infrastructure, cloud, containers, orchestration, and distributed systems. ### Source excerpt Expedia Group Technology -- EngineeringA system that facilitates investigation of service degradations and outages using service telemetry data and AIPhoto by Evangelos Mpikakis on Unsplash. The recent advancements in the artificial intelligence space make us re-evaluate how work is done. From programming, to designing systems, or even operating them in production. While there is considerable focus on automating programming, one area which could undergo transformation is how we monitor and operate our systems and services. A few of us came together and designed Expedia's® Service Telemetry Analyzer (STAR), an early iteration of a system that facilitates investigation of service degradations and outages using service telemetry data and AI models and techniques. Expedia's Service Telemetry Analyzer (STAR) The early product offering includes: Execution of multi-step workflows. Integration of software and systems engineering knowledge, including application and infrastructure, cloud, containerization, and orchestration patterns, into diagnostic workflows for complex distributed systems. Application of domain-specific prompt engineering for metric and root cause analysis. Utilization of advanced off-the-shelf AI models. Implementation of prompt engineering techniques, including role prompting, prompt chaining, and generated knowledge prompting. Design The product offering is a web-based service that provides an application programming interface (API). While AI agents and chatbots are gaining traction, we aimed to start with something a) simple, b) precise (to a certain extent, considering the potential hallucinations of the models), and c) that avoids the additional and currently less understood failure modes of an agent. As this field evolves, we will continue to iterate on the design. Therefore, there is limited context engineering beyond domain-specific prompts; for instance, there is no support for function calling / tool use, short-term and long-term memory, or retri ## Scaling Whatnot: Behind the Largest Live Shopping Stream in US History DevFeed: [Scaling Whatnot: Behind the Largest Live Shopping Stream in US History]() Original publisher: [Read original article]() Author: Whatnot Engineering Published: 2026-02-24T14:33:16Z Content type: article Language: en Sources: [Whatnot Engineering]() Topics: [Scalability](), [Architecture & Design](), [Platform Engineering](), [SRE](), [Elixir](), [Python]() Tags: [cloud-computing](), [devops](), [elixir](), [infrastructure](), [load-testing](), [python](), [resilience](), [scalability](), [scale](), [site-reliability-engineer](), [software-engineering](), [sre](), [technology]() ### AI overview Whatnot describes how it prepared its platform for a MrBeast giveaway stream that reached 583,000 concurrent viewers and became the largest live shopping event in US history. The article covers architectural investments, progressive production load testing, event-day results, and lessons for future scalability. ### Source excerpt On February 8, 2026, over a half million viewers tuned in to watch MrBeast give away 1 million dollars in prizes on Whatnot. On Big Game Sunday 2026, MrBeast went live on Whatnot for a giveaway show that would become the largest live shopping event in US history. At peak, 583,000 concurrent viewers were watching a single show on our platform. Over 555k people entered a single giveaway. We drove hundreds of thousands of new signups in 24 hours. If any one of a dozen systems buckled, it would have happened live on camera. We pulled it off with zero major incidents. But that outcome was never guaranteed. It took months of preparation, 60+ engineers across every major engineering org, and some of the most significant infrastructure investments we've ever made. In this post, we'll walk through the biggest technical challenges we faced and how we solved them, not with throwaway scaffolding, but with durable platform improvements that raise our scalability ceiling for every seller and buyer on the platform. We'll cover the work in three parts. First, the key architectural investments we made to handle this scale: admission control, connection pooling, feed resilience, and video infrastructure. Then, how we validated it all through progressive production load testing. Finally, what happened on event day, what we learned, and what we're carrying forward. Setting the Stage If you've followed our blog, you might remember our Post Malone "Post-Poned" post from 2022 or our three-part series on preparing for the 2024 Big Game. Each of those events pushed us to improve, and each one revealed new limits. As our community has grown, scaling our infrastructure to match has been a consistent priority. The MrBeast event was on a different order of magnitude entirely, but it accelerated work that was already underway. Our target was to support 1 million concurrent viewers on a single stream and 1.35 million across the platform. To put that in perspective, our previous largest event had ## Built to Scale: How Nexxiot Delivers Always-On Global IoT with CockroachDB DevFeed: [Built to Scale: How Nexxiot Delivers Always-On Global IoT with CockroachDB]() Original publisher: [Read original article]() Author: David Weiss Published: 2025-05-30T00:00:00Z Content type: article Language: en Sources: [Cockroach Labs]() Topics: [CockroachDB](), [Databases](), [Internet of things](), [SQL](), [Availability](), [consistency](), [Resilience](), [DynamoDB](), [elasticsearch](), [Amazon RDS](), [upgrade](), [site-reliability-engineer]() Tags: [amazon-rds](), [availability](), [cloud](), [cockroachdb](), [consistency](), [database](), [databases](), [devices](), [distributed](), [dynamodb](), [elasticsearch](), [iot](), [resilience](), [site-reliability-engineer](), [sql](), [upgrade]() ### AI overview The article describes how Nexxiot selected CockroachDB for a global transport operations platform using solar-powered IoT devices. It explains that the company moved beyond DynamoDB and Elasticsearch as its workloads grew, citing concerns about transactional consistency, query complexity, scalability, availability, and operational overhead. ### Source excerpt When you're building a transport operations platform to optimize global fleets, "good enough" isn't good enough. ## In Focus: Blinkit SRE Engineer Sumanth Reddy on Resilient Software and Scalable Database Migrations DevFeed: [In Focus: Blinkit SRE Engineer Sumanth Reddy on Resilient Software and Scalable Database Migrations]() Original publisher: [Read original article]() Author: Jacob Published: 2023-03-16T03:21:01Z Content type: article Language: en Sources: [Grofers]() Topics: [Resilience](), [SRE](), [DevOps](), [Database]() Tags: [culture](), [database](), [devops](), [engineering](), [people-at-blinkit](), [quick-commerce](), [resilience](), [site-reliability-engineer](), [sre]() ### AI overview An interview with Blinkit SDE 3 Chinthakunta Sumanth Kumar Reddy about his journey, work in Software Resilience Engineering, scalable database migrations, resilient software, and challenges in the instant commerce space. ### Source excerpt A conversation with engineers who help run Blinkit Chinthakunta Sumanth Kumar Reddy is an SDE 3 at Blinkit. He joined us in March 2021 and has since helped us build a resilient application platform at Blinkit. He currently works as a part of Software Resilience Engineering (SRE)-enabling scalable database migrations for Blinkit's applications. Tell us about your background and your journey in Blinkit so far. I have always been curious to learn new things and don't have barriers as long as I can engage with the task. Hailing from a small village in Andhra Pradesh, I have travelled across several cities (Fun fact: My village and family share the same name). I have been working with the SRE team for almost two years. I love the no-nonsense work culture here, which allowed me to raise PR from day 2 in the company. Why did you choose to work in tech? What drove you? Due to my background in the Electronics branch, I have always been fascinated by how the physical transistors click and clack with seemingly intangible software bits. There is a sense of excitement when things spring into action, and I think if there's an equivalent of magic in our world, it will be Tech. What excites you the most about your field? The ability to build resilient software excites me. Imagine seeing a traffic spike and everything scaling up to take the load and winding down when the traffic calms down, all on its own-It's magical, to say the least. While solving a problem, what is the principle you always adhere to? When hit with a problem, take a step back and think through it. Sometimes, it isn't what it appears to be. Problem-solving is all about perspectives. What immediate challenges are we trying to solve in the instant commerce space? Efficiency and localisation. With delivery timelines moving from days to minutes, there is significantly lesser room for errors. Also, since we are nearer to customers, localisation is the key. For example, a top seller in one area might be utterly irreleva ## In Focus: Jay Dihenkar DevFeed: [In Focus: Jay Dihenkar]() Original publisher: [Read original article]() Author: Jacob Published: 2023-02-23T03:01:29Z Content type: opinion Language: en Sources: [Grofers]() Topics: [release engineering](), [SRE](), [CI/CD](), [Resilience](), [Monitoring](), [observability](), [developer-productivity](), [Automation](), [Scalability](), [systems]() Tags: [automation](), [ci-cd](), [culture](), [developer](), [developer-productivity](), [devops](), [monitoring](), [observability](), [people-at-blinkit](), [quick-commerce](), [release-engineering](), [resilience](), [scalability](), [site-reliability-engineer](), [sre](), [systems]() ### AI overview An interview with Blinkit Staff Engineer Jay Dihenkar about his journey from release engineering to site reliability engineering. He discusses CI/CD, developer productivity, production incident debugging, automation, monitoring and alerting, infrastructure costs, application reliability patterns, and challenges in scaling quick-commerce logistics. ### Source excerpt A conversation with engineers who help run Blinkit Jay Dihenkar is a Staff Engineer at Blinkit. He joined us in December 2020 and has helped different teams manage and streamline their build and release processes. He is currently working towards continuously improving the reliability, scalability, observability, developer productivity, and other such aspects of a software system critical for ensuring that the system can meet the needs of its users and stakeholders over time. Tell us something about yourself and your journey in Blinkit so far. I started out as an engineer on the Release Engineering team, working on goals of Software Quality/Stability and streamlining release processes (Continuous Integration/Continuous Deployment or CI/CD). I help make an impact by improving developer productivity while safeguarding the production ecosystem against failures. I later transitioned into the Site Reliability Engineering (SRE) team, where we focused on improving the reliability and resilience of our systems. Being a site-reliability engineer requires one to develop a deep understanding of how the various components of a system work together and how to optimize them. Overall my journey in Blinkit is one of continuous learning and improvement. If you were to describe your experience as a Blinker in one word, what would that be? Growth. Being here is all about continuous learning in a fast-paced environment. What does a typical day at work look like for you? On a typical day, I work on the following domains (though not everything on a single day!): debugging complex problems sprung out of production incidents, putting automation in places where required, setting up monitoring and alerting, as well as responding to alerts, release engineering and developer productivity (CI/CD), System Infrastructure Cost, Working toward implementing Application Reliability Patterns and so on... What are the biggest challenges facing quick commerce today? I believe the "Logistics infrastructure" ## Sponsored Post: G-Core Labs, Close, Wynter, Pinecone, Kinsta, Bridgecrew, IP2Location, StackHawk, InterviewCamp.io, Educative, Stream, Fauna, Triplebyte DevFeed: [Sponsored Post: G-Core Labs, Close, Wynter, Pinecone, Kinsta, Bridgecrew, IP2Location, StackHawk, InterviewCamp.io, Educative, Stream, Fauna, Triplebyte]() Original publisher: [Read original article]() Author: HighScalability Team Published: 2022-06-10T15:39:03Z Content type: article Language: en Sources: [High Scalability](), [High Scalability]() Topics: [DevOps](), [site-reliability-engineer](), [Databases](), [Monitoring](), [Ansible](), [hosting](), [Linux](), [PHP](), [JavaScript](), [NoSQL](), [WordPress](), [Ubuntu]() Tags: [ansible](), [databases](), [debian](), [devops](), [hosting](), [javascript](), [linux](), [lxc](), [monitoring](), [nosql](), [php](), [site-reliability-engineer](), [sponsored](), [sponsored-post](), [ubuntu](), [wordpress]() ### AI overview A sponsored post presents hiring and promotional opportunities from several technology companies, including Site Reliability Engineer, DevOps Engineer, SysOps Engineer, and JavaScript Developer roles. It highlights experience with databases, infrastructure, backend services, Linux, PHP, Ansible, LXC, hosting, WordPress, and JavaScript. ### Source excerpt Who's Hiring? Close is building the sales communication platform of the future and we're looking for a Site Reliability Engineer to help us accomplish that goal. If you have a passion for databases (both SQL and NoSQL), significant experience building, managing, and monitoring infrastructure, databases, and backend services at scale and want to work with a rag-tag team of hardworking, but humble humans, then come check us out! Wynter is looking for system administrators, engineers, and developers to join its research panel. Participate in research surveys, get paid ($90-$180/hr) for your feedback and comments. Super low key commitment, 10-15 mins per survey. Learn more and sign up. DevOps Engineer: At Kinsta, we set out to create the best managed hosting platform in the world. If you are an experienced DevOps Engineer who is constantly looking for ways to innovate and improve, we might just be the place for you! As Kinsta's DevOps Engineer, you will be instrumental in making sure that our infrastructure is always on the bleeding edge of technology, remaining stable and high-performing at all times. If you love working with Linux, have a background in PHP, and have worked with cutting-edge technologies, such as Ansible and LXC, check us out and apply here. SysOps Engineer: As Kinsta's SysOps Engineer, you'll play a key role in caring for the health of our servers, preventing issues, and responding immediately to mitigate any problems in our infrastructure. If you have experience in hosting and with the WordPress stack, have worked with Ubuntu or Debian-based systems, and cutting-edge technologies, such Ansible and LXC, you should definitely check us out and apply here! JavaScript Developer: Kinsta's Development Team aims to create truly effective solutions for our customers and our internal teams. We believe the only way to maintain and scale our standards is to focus on quality code. If you are a talented JavaScript developer and you would like to work with an owner ## Developer Friendly Solid Infrastructure Security DevFeed: [Developer Friendly Solid Infrastructure Security]() Original publisher: [Read original article]() Author: info@goteleport.com (Ev Kontsevoy) Published: 2020-05-06T00:00:00Z Content type: article Language: en Sources: [Teleport]() Topics: [Security](), [cloud-infrastructure](), [Architecture & Design](), [site-reliability-engineer]() Tags: [ansible](), [cloud](), [complexity](), [downtime](), [infrastructure](), [jenkins](), [patterns](), [security](), [site-reliability-engineer](), [ssh]() ### AI overview This article examines how SaaS companies balance strong cloud infrastructure security with developer productivity. It discusses the costs of excessive security controls, including workarounds such as unauthorized bastions and additional services on bastion hosts, and argues that unnecessary complexity can increase security risk and human error. ### Source excerpt How SaaS companies approach the trade-off between having solid cloud infrastructure security and upsetting their own engineers by overdoing it ## I'm Joining TOTVS Labs DevFeed: [I'm Joining TOTVS Labs]() Original publisher: [Read original article]() Author: Carlos Alexandro Becker Published: 2018-04-17T00:00:00Z Content type: opinion Language: en Sources: [Carlos Becker]() Topics: [site-reliability-engineering](), [Kubernetes clusters](), [Monitoring](), [CI/CD](), [reliability]() Tags: [availability](), [ci-cd](), [kubernetes](), [kubernetes-clusters](), [monitoring](), [site-reliability-engineer]() ### AI overview The author announces leaving ContaAzul after four years to join TOTVS Labs as a remote Site Reliability Engineer. The post reflects on work involving electronic invoicing, Kubernetes clusters, Prometheus monitoring, CI/CD pipeline improvements, and availability improvements. ### Source excerpt Everything in life eventually comes to an end, including life itself. This is not that kind of post, though.