# downtime Published articles for downtime. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Investigating Redis Client Failures After a Node Loss at hh.ru DevFeed: [Investigating Redis Client Failures After a Node Loss at hh.ru]() Original publisher: [Read original article]() Author: nicholasgribanov (hh.ru) Published: 2026-08-26T05:30:49Z Content type: article Language: ru Sources: [HeadHunter RU]() Topics: [Redis]() Tags: [consul](), [downtime](), [java](), [jedis](), [redis](), [sre](), [tag-1d4e49052118](), [tag-68e701e78517](), [tag-b0a411324cb6](), [tag-b92bf5906bbd](), [tag-dfac9042ce7b](), [timeout]() ### AI overview A Platform team at hh.ru investigates incidents in its internal Redis client, including a 57-minute outage after a Redis node was lost. The team used controlled-load testing, Jedis updates, timeout and pool experiments, new metrics, and logs to trace the failures to a code issue and improve client stability. ### Source excerpt Пару лет назад мы устроили настоящее расследование серии инцидентов в поисках скрытого дефекта нашего Redis-клиента. Команда воспроизводила сбои под контролируемой нагрузкой, проверяла одну гипотезу за другой, обновляла Jedis, экспериментировала с таймаутами и размерами пулов -- но ничего не помогало. А помогли новые метрики и логи, настойчивость команды, ночные эксперименты и готовность разбирать поведение системы до последнего соединения. Получилась история с неожиданными поворотами, ложными следами и одной лишней строчкой кода в роли главного подозреваемого -- а её итогом стал Redis-клиент, который оказался устойчивее, чем был до начала расследования. Меня зовут Коля Грибанов, я тимлид команды "Платформа" в hh.ru. В статье расскажу, почему потеря одной ноды Redis вызывала шторм из десятков тысяч соединений, и как мы шаг за шагом искали причину инцидентов. Читать далее ## Five Deployment Strategies for Safer Software Releases DevFeed: [Five Deployment Strategies for Safer Software Releases]() Original publisher: [Read original article]() Author: Saurabh Dashora Published: 2026-05-12T07:05:17Z Content type: article Language: en Sources: [System Design Codex]() Topics: [Deployment Strategies](), [Deployment](), [progressive delivery](), [App]() Tags: [blue-green-deployment](), [deployment-strategies](), [downtime](), [load-balancer](), [progressive-delivery](), [rollback]() ### AI overview This article discusses deployment strategies for reducing release risk, downtime, and user impact. It explains blue/green deployment and canary deployment, including their workflows, benefits, tradeoffs, and suitable use cases, and introduces A/B testing as part of a release process. ### Source excerpt Every developer should know these ## Zero downtime Upgrade: Yelp's Cassandra 4.x Upgrade Story DevFeed: [Zero downtime Upgrade: Yelp's Cassandra 4.x Upgrade Story]() Original publisher: [Read original article]() Author: Mark Surnin and Muhammad Junaid Muzammil, Software Engineer Published: 2026-04-07T00:00:00Z Content type: article Language: en Sources: [Yelp]() Topics: [Apache Cassandra](), [upgrade](), [NoSQL](), [Database](), [observability](), [Kubernetes]() Tags: [2026](), [apache](), [cassandra](), [clusters](), [downtime](), [kubernetes](), [nosql](), [observability](), [reliability-engineering](), [upgrade]() ### AI overview Yelp's Database Reliability Engineering team describes upgrading more than a thousand Cassandra nodes from 3.11 to 4.1 on Kubernetes without downtime. The article covers the motivation, expected reliability and performance improvements, operational guardrails, certificate handling, repairs, logging, and compatibility work for related components. ### Source excerpt The Database Reliability Engineering team at Yelp seamlessly upgraded more than a thousand Cassandra nodes with zero downtime. This post takes you behind the scenes of our upgrade strategy, from planning sessions to flawless rollouts. Background Motivation Apache Cassandra is a distributed wide-column NoSQL datastore and is used widely at Yelp for storing both primary and derived data. Yelp orchestrates Cassandra clusters on Kubernetes with the help of operators, as explained in our operator overview post. Upgrading from Cassandra 3.11 to 4.1 offered several observability and reliability improvements, in addition to performance gains. Based on public benchmarks, we expected to... ## Operating Trino at Scale With Trino Gateway DevFeed: [Operating Trino at Scale With Trino Gateway]() Original publisher: [Read original article]() Author: Prakhar Sapre Published: 2026-03-24T12:01:00Z Content type: article Language: en Sources: [Expedia]() Topics: [gateway](), [Load Balancing](), [Routing (disambiguation)](), [data analytics](), [Authentication](), [SQL]() Tags: [analytics](), [authentication](), [clusters](), [concurrency](), [data-science](), [downtime](), [gateway](), [load-balancer](), [load-balancing](), [open-source](), [sql](), [trino](), [trino-gateway](), [trinos]() ### AI overview This article explains how Trino Gateway routes queries across multiple Trino clusters and centralizes routing, authentication, load balancing, monitoring, and cluster management. It describes the project's origins as Presto Gateway at Lyft and its role in supporting larger analytics platforms with more complex workloads and higher concurrency. ### Source excerpt Expedia Group Technology -- DataWorkload-aware routing for TrinoPhoto by Joseph Barrientos on Unsplash Trino -- a fork of PrestoSQL -- is a powerful tool in modern data analytics, enabling organizations to query large datasets quickly and efficiently. As a distributed SQL query engine, Trino provides fast, scalable insights without requiring data relocation. While Trino is robust on its own, its capabilities are further enhanced when paired with a Gateway, which introduces features such as query routing, strong security, and streamlined cluster management. A brief overview The Gateway project originated at Lyft as Presto Gateway, serving as a proxy and load balancer for PrestoDB. It was later forked and integrated into the Trino ecosystem, with contributions from various organizations and the open-source community. The Gateway serves as a central point for managing and routing queries, providing a unified interface for users and administrators. As organizations scale their analytics platforms, they often encounter challenges such as increased query complexity, higher concurrency, and the need for specialized cluster configurations. Directing users to specific cluster endpoints becomes impractical as the user base grows. A Gateway addresses these challenges by routing queries to the most appropriate clusters based on workload, improving efficiency and responsiveness. The Gateway acts as a vital intermediary between users and the Trino query engine. By abstracting the complexities of distributed query execution, it manages critical functions such as routing, authentication, and load balancing across diverse backend clusters. This ensures that queries are efficiently directed to the optimal processing cluster. With an intuitive user interface, the Gateway transforms what was once a convoluted process into a manageable and transparent experience empowering administrators with real-time insights and precise control over their backend cluster infrastructure. Whether it's mon ## System Migration: Minimize Downtime, Maximize Efficiency DevFeed: [System Migration: Minimize Downtime, Maximize Efficiency]() Original publisher: [Read original article]() Author: hello@ankit-rana.com Published: 2026-03-16T00:00:00Z Content type: tutorial Language: en Sources: [Ankit Rana | Mechanical Sympathy]() Topics: [migration](), [systems](), [async](), [client](), [event driven](), [Kafka](), [API]() Tags: [apache-kafka](), [api](), [architecture](), [async](), [bridge-layer](), [data-pipelines](), [downtime](), [event-driven-architecture](), [metrics](), [migration](), [observability](), [production](), [reliability](), [rollback](), [system-migration](), [traffic-leakage]() ### AI overview A practical guide to migrating an existing system with minimal disruption. It recommends isolated-environment testing, load testing, adapters for incompatible contracts, synchronized asynchronous pipelines, a Kafka-based shared stream, a bridge layer, staged traffic switching, monitoring, and rollback preparation. ### Source excerpt Migrate behind a bridge layer that routes all client traffic and supports three modes: old-only, dual, and new-only. Run dual mode to compare responses without user impact, keep a back-sync pipeline so the old system stays current for rollback, and shift traffic in stages while watching metrics at each step. ## Moving Mountains of Data off S3 DevFeed: [Moving Mountains of Data off S3]() Original publisher: [Read original article]() Author: Jeremy Daer, Kimberly Rhodes, Fernando Olivares Published: 2026-01-08T18:00:00Z Content type: article Language: en Sources: [37signals Dev]() Topics: [Amazon S3](), [migration](), [data](), [amazon](), [Rails]() Tags: [amazon-s3](), [downtime](), [go](), [migration](), [s3]() ### AI overview A RECORDABLES episode explains how 37signals moved billions of files, including about five billion objects across five petabytes of data, out of Amazon S3 without downtime. It discusses bandwidth limits, AWS constraints, custom Rails tooling, verification, and the risks of deleting the original data. ### Source excerpt Principal Programmer Jeremy Daer explains how we moved billions of files out of Amazon S3 with zero downtime. ## How banks are cutting development time in half while avoiding million-dollar outages with Temporal Cloud DevFeed: [How banks are cutting development time in half while avoiding million-dollar outages with Temporal Cloud]() Original publisher: [Read original article]() Author: Hannah Short Published: 2025-09-09T00:00:00Z Content type: article Language: en Sources: [Temporal Blog]() Topics: [Cloud](), [Development](), [reliability](), [distributed-systems]() Tags: [announcements](), [banking](), [cloud](), [distributed-systems](), [downtime](), [financial-services](), [productivity](), [reconciliation](), [reliability]() ### AI overview A Forrester Total Economic Impact study of Temporal Cloud describes banking organizations replacing homegrown workflow systems, with reported improvements including $14.3 million in preserved revenue over three years and feature delivery in half the time. ### Source excerpt Forrester's TEI study reveals Temporal Cloud delivers $14.3M in preserved revenue and 50% faster feature delivery for banking institutions. ## Leveraging Change Data Capture For Database Migrations At Scale DevFeed: [Leveraging Change Data Capture For Database Migrations At Scale]() Original publisher: [Read original article]() Author: Pravish Sood Published: 2025-05-13T20:30:01Z Content type: article Language: en Sources: [Squarespace](), [Squarespace Engineering Blog]() Topics: [Databases](), [migration](), [CockroachDB](), [PostgreSQL](), [systems]() Tags: [cockroachdb](), [database](), [distributed](), [downtime](), [migrations](), [postgresql](), [scale]() ### AI overview Squarespace describes migrating databases from PostgreSQL to CockroachDB to improve horizontal scalability and reduce downtime and operational risk. The Databases team developed a migration plan and in-house tooling to handle schema, user, and data migration across varied workloads. ### Source excerpt Our teams love relational databases and the benefits ACID properties bring to us as engineers, but they have their limits, especially in horizontal scaling. We had to adopt new database tech across a fleet of highly available services while eliminating as much risk as possible... but how? Enter: change data capture patterning. ## Understanding fault tolerance in distributed systems DevFeed: [Understanding fault tolerance in distributed systems]() Original publisher: [Read original article]() Author: Lauren Bennett Published: 2025-01-08T00:00:00Z Content type: article Language: en Sources: [Temporal Blog]() Topics: [distributed-systems](), [systems](), [Availability](), [Cloud](), [Replication](), [backups]() Tags: [availability](), [backups](), [cloud](), [data](), [distributed](), [distributed-systems](), [downtime](), [failover](), [fault-tolerance](), [hardware](), [outage](), [replication](), [services](), [software](), [systems](), [temporal-concepts](), [user]() ### AI overview An explanation of fault tolerance in distributed systems, including how it keeps systems operating during failures and limits disruptions, downtime, and data loss. It describes redundancy, replication, failover mechanisms, graceful degradation, and the relationship between high availability and fault tolerance. ### Source excerpt Discover what fault tolerance is and how it ensures reliable systems with key principles and examples in cloud environments. ## Kubernetes Namespace Restriction and Separation DevFeed: [Kubernetes Namespace Restriction and Separation]() Original publisher: [Read original article]() Author: jackson.pitts@goteleport.com (Jack Pitts) Published: 2024-10-03T00:00:00Z Content type: tutorial Language: en Sources: [Teleport]() Topics: [Kubernetes](), [Kubernetes clusters](), [Security](), [Requirements](), [Deployment]() Tags: [compliance](), [deployment](), [downtime](), [financial-services](), [kubernetes](), [kubernetes-clusters](), [requirements](), [security](), [standards]() ### AI overview This article explains how Teleport can help financial services organizations restrict and separate Kubernetes namespaces. It discusses namespace-based resource and workload isolation, environment segmentation, resource quotas, compliance requirements, and improved engineering productivity. ### Source excerpt Learn how Teleport enables secure and scalable namespace restriction and separation in Kubernetes clusters. ## Migrating Kafka topics without downtime DevFeed: [Migrating Kafka topics without downtime]() Original publisher: [Read original article]() Published: 2022-09-15T00:00:00Z Content type: tutorial Language: en Sources: [Gaurav Sarma's Blog]() Topics: [Kafka](), [ordering](), [Scalability](), [version]() Tags: [downtime](), [kafka](), [ordering](), [partition](), [versioning]() ### AI overview A tutorial for migrating Kafka topics by creating versioned replacement topics, redirecting publishers and consumers, and draining the old topics. It explains how partition counts affect ordering, concurrency, scalability, and management overhead, and notes that the author did not find a direct tool for migration without downtime. ### Source excerpt Each kafka topic defines the number of partitions and replication factors when it's created. However, once a topic is created, the partition count cannot be changed without affecting the ordering guarantees of the kafka partitions since kafka uses the following formula to calculate which partition a record should go to: partition\_id = partition\_key % number of partitions Kafka partitions are the gateway to concurrency and scalability... ## LtU is now running in a new, more stable environment DevFeed: [LtU is now running in a new, more stable environment]() Original publisher: [Read original article]() Published: 2022-08-31T06:37:45Z Content type: news Language: en Sources: [Lambda the Ultimate]() Topics: [Software](), [coding]() Tags: [admin](), [downtime](), [infrastructure](), [running](), [software]() ### AI overview LtU experienced an extended period of downtime because its software infrastructure had become outdated and difficult to maintain. It has since migrated to a new environment expected to improve stability. ### Source excerpt LtU has experienced a long period of downtime recently. Its software infrastructure was outdated enough that it became difficult to maintain when problems arose. It has now been migrated to a brand new environment. It should be much more stable from now on. ## Cluster build log: Moving to temporary cluster DevFeed: [Cluster build log: Moving to temporary cluster]() Original publisher: [Read original article]() Published: 2021-09-01T00:00:00Z Content type: article Language: en Sources: [Community blog on PINE64]() Topics: [hosting](), [Network](), [Ethernet]() Tags: [cluster](), [community](), [downtime](), [ethernet](), [hosting](), [network](), [rockpro64-2]() ### AI overview This build log explains why PINE64 is creating a temporary hosting cluster. The existing cluster needs hardware changes that require extended downtime, so a smaller cluster will keep community services running during the work. The article introduces the power and network requirements for the temporary cluster. ### Source excerpt As promised in the last month's community update Housekeeping section, this is the first part of the cluster's build log! This mini-series in which I will cover everything about upcoming changes our hosting cluster, and it begins with building a temporary cluster. Why build a temporary cluster? One of the latest photo of cluster Let's start from beginning. When we built the current hosting cluster a year ago, it was made in quite a hurry (some of the services needed to be migrated to the cluster on a moment's notice). Also, it was the first time I (gamiee) and Matthew (fire219) were tasked with building a cluster. Because of this, some of the cluster's features are missing and various things don't work quite as well as we have hoped. For example, the cluster has power management boards (PCB on right side of every row), which are used primary for power distribution to the SBCs, but also it also has a I2C GPIO Expander for rebooting and turning off the SBCs. The I2C Expansion isn't in use yet, because the primary node wasn't able to detect it, so this is something that needs to be checked out when the cluster is turned off. And this is exactly the main reason why we are building a temporary cluster. We need to do some changes which require us to have the whole cluster turned off and on a workbench (not in housing room where isn't much space), and this means that it will be inaccessible for several days. Obviously a long downtime wouldn't be good when 100% of PINE64 community services are running on it. So we are going to build a temporary small cluster, which will have the minimal number of required nodes to have all services running, so we can make all upgrades with precision and at our own pace. What do we need to make a custom cluster? There are two things each node needs: a power supply and access to the network. Access to the network can be solved by using a classic Ethernet switch, but powering multiple RockPros, which are actually pretty power hungry, is a bit ## Atomic Deployments Without Tears DevFeed: [Atomic Deployments Without Tears]() Original publisher: [Read original article]() Author: andrew@nystudio107.com (Andrew Welch) Published: 2020-06-30T04:00:00Z Content type: tutorial Language: en Sources: [nystudio107 | Articles on modern web development.]() Topics: [CI/CD](), [Deployment](), [Git](), [Content Management System](), [Development]() Tags: [amazon-s3](), [atomic](), [automatically](), [changes](), [continuous](), [continuous-integration](), [deploy](), [deployments](), [downtime](), [git](), [how-to](), [insights](), [integration](), [learn](), [safely](), [tools](), [using](), [website](), [zero]() ### AI overview This tutorial explains how to use Continuous Integration tools to perform atomic, zero-downtime deployments for Craft CMS websites. It covers a workflow based on local development, a cloud Git repository, and automated deployment to production or staging servers. ### Source excerpt Learn how to use atomic deployments to automatically and safely deploy changes to your website with zero downtime using Continuous Integration (CI) tools ## Developer Friendly Solid Infrastructure Security DevFeed: [Developer Friendly Solid Infrastructure Security]() Original publisher: [Read original article]() Author: info@goteleport.com (Ev Kontsevoy) Published: 2020-05-06T00:00:00Z Content type: article Language: en Sources: [Teleport]() Topics: [Security](), [cloud-infrastructure](), [Architecture & Design](), [site-reliability-engineer]() Tags: [ansible](), [cloud](), [complexity](), [downtime](), [infrastructure](), [jenkins](), [patterns](), [security](), [site-reliability-engineer](), [ssh]() ### AI overview This article examines how SaaS companies balance strong cloud infrastructure security with developer productivity. It discusses the costs of excessive security controls, including workarounds such as unauthorized bastions and additional services on bastion hosts, and argues that unnecessary complexity can increase security risk and human error. ### Source excerpt How SaaS companies approach the trade-off between having solid cloud infrastructure security and upsetting their own engineers by overdoing it ## How to Create an Index in Django Without Downtime DevFeed: [How to Create an Index in Django Without Downtime]() Original publisher: [Read original article]() Author: Haki Benita Published: 2019-04-09T21:00:00Z Content type: tutorial Language: en Sources: [Haki Benita]() Topics: [Django](), [migration](), [SQL]() Tags: [articles](), [atomic](), [django](), [downtime](), [how-to](), [migration](), [migrations](), [orm](), [postgresql](), [sql]() ### AI overview A tutorial on creating a Django database index without downtime. It explains graceful migrations, atomic and reversible migrations, executing raw SQL in migrations, and changing Django's built-in migration behavior. ### Source excerpt If you ever had to maintain a traffic heavy Django site, you probably had to deal with graceful migrations. In the article I explain what atomic and reversible migrations are, how to execute "raw" SQL in migrations the right way, and how using a little known migration command we can completely alter the Django migrations built-in behavior. ## Managing db schema changes without downtime DevFeed: [Managing db schema changes without downtime]() Original publisher: [Read original article]() Author: Sam Saffron Published: 2018-03-22T06:30:05Z Content type: article Language: en Sources: [Sam Saffron]() Topics: [Database Migration](), [migration](), [Deployment](), [Rails](), [CI/CD]() Tags: [activerecord](), [commit](), [continuous-deployment](), [database](), [deployment](), [downtime](), [exceptions](), [integration-test](), [outage](), [schema]() ### AI overview This article explains how database schema changes can cause deployment outages, especially when older application instances remain active and ActiveRecord caches schema information. It describes Discourse's use of richer migration logging and deployment patterns to reduce these risks. ### Source excerpt At Discourse we have always been huge fans of continuous deployment. Every commit we make heads to our continuous integration test suite. If all the tests pass (ui, unit, integration, smoke) we automatically deploy the latest version of our code to https://meta.discourse.org. This pattern and practice we follow allows the thousands of self-installers out there to safely upgrade to the tests-passed version whenever they feel like it. Because we deploy so often we need to take extra care not to have any outages during deployments. One of the most common reasons for outages during application deployment is database schema changes. The problem with schema changes Our current deployment mechanism roughly goes as follows: Migrate database to new schema Bundle up application into a single docker image Push to registry Spin down old instance, pull new instance, spin up new instance (and repeat) If we ever create an incompatible database schema we risk breaking all the old application instances running older versions of our code. In practice, this can lead to tens of minutes of outage! In ActiveRecord the situation is particularly dire cause in production the database schema is cached and any changes in schema that drop or rename columns very quickly risk breaking every query to the affected model raising invalid schema exceptions. Over the years we have introduced various patterns to overcome this problem and enable us to deploy schema changes safely, minimizing outages. Tracking rich information about migrations ActiveRecord has a table called schema_migrations where it stores information about migrations that ran. Unfortunately the amount of data stored in this table is extremely limited, in fact it boils down to: connection.create_table(table_name, id: false) do |t| t.string :version, version_options end The table has a lonely column storing the "version" of migrations that ran. It does not store when the migration ran It does not store how long it took the migration to ## How trivago Reduced Memcached Memory Usage by 50% DevFeed: [How trivago Reduced Memcached Memory Usage by 50%]() Original publisher: [Read original article]() Author: Ivana Petrovic Software backend developer; Optimisation; Clean Architecture Published: 2017-12-19T00:00:00Z Content type: article Language: en Sources: [Trivago]() Topics: [Caching](), [Code](), [PHP](), [Logging](), [Network](), [Database]() Tags: [backend](), [cache](), [caching](), [downtime](), [errors](), [high-performance](), [logging](), [memcached](), [memory](), [network](), [performance](), [php](), [profiling](), [uptime]() ### AI overview trivago describes how a Memcached cache failure caused database overload and platform downtime during a botnet-driven spike in spider traffic. The team added cache-value-size logging, used Blackfire profiling, and found that a method was caching about 10 MB of data and being called 30 times per page load. ### Source excerpt One day, Memcached ran out of free memory. The method `get` failed and all requests went directly to the database. Of course these calls also failed under the huge load, and eventually it caused downtime for the whole trivago website. Yikes! ## Better database migrations in Postgres DevFeed: [Better database migrations in Postgres]() Original publisher: [Read original article]() Author: Map Published: 2017-09-10T20:55:56Z Content type: tutorial Language: en Sources: [Craig Kerstiens]() Topics: [Database](), [Databases](), [Development]() Tags: [database](), [downtime](), [migrations](), [postgres](), [postgresql](), [production]() ### AI overview This article explains how Postgres schema migrations can affect production as databases grow. It focuses on adding columns, constraints, default values, and indexes, and recommends staged migrations with gradual backfilling to reduce downtime. ### Source excerpt As your database grows and scales there are some operations that you need to take more care of than you did when you were just starting. When working with your application in your dev environment you may not be fully aware of the cost of some operations until you run them against production. And at some point most of us have been guilty of it, running some migration that starts at 5 minutes, then 15 minutes in it's still running, and suddenly production traffic is impacted. There are two operations that tend to happen quite frequently, each with some straightforward approaches to mitigate having any noticable amount of downtime. Let's look at each of the operations, how they work and then how you can approach them in a safer way. Adding new columns Adding a new column is actually quite cheap in Postgres. When you do this it updates its underlying tracking of the columns that exist-which is almost instant. The part that becomes expensive is when you have some constraint against the column. A constraint could be a primary or foreign key, or some uniqueness constraint. Here Postgres has to scan through all the records in the table to ensure that it's not being violated. Adding some constraint such as not null does happen some, but is not the most common cause. The most common reason for slowness of adding a new column is that most frameworks make it very simple for you to set a default value for the new column. It's one thing to do this for all new records, but when you do this when an existing table it means the database has to read all the records and re-write them with the new default value attached. This isn't so bad for a table with a few hundred records, but for a few hundred million run it then go get yourself coffee, or lunch, or a 5 course meal because you'll be waiting for a while. In short, not null and setting a default value (on creation) of your new column will cause you pain. The solution is to not do those things. But, what if you want to have a default ## Localytics' Process for Responding to Service Incidents DevFeed: [Localytics' Process for Responding to Service Incidents]() Original publisher: [Read original article]() Author: Tony Wieczorek Published: 2016-09-13T18:24:39Z Content type: article Language: en Sources: [Localytics]() Topics: [incident](), [systems](), [Slack](), [realtime]() Tags: [downtime](), [engineering](), [incident](), [pagerduty](), [postmortem](), [realtime](), [slack](), [systems]() ### AI overview Localytics describes its process for responding to service degradations and downtime. The process uses Slack and PagerDuty to coordinate triage, debugging, incident follow-up, postmortems, and communication with company leaders and customers. ### Source excerpt We believe the highest performing engineering teams have a process to identify, triage, fix and learn from service degradations and downtime. At Localytics, we build highly available and scalable systems, and part of the key to our success is learning from failures. One way we foster a learning culture is ## MySQL Migrations DevFeed: [MySQL Migrations]() Original publisher: [Read original article]() Author: Zoli Kahan Published: 2016-08-27T05:00:00Z Content type: tutorial Language: en Sources: [Zolmeister]() Topics: [MySQL](), [migration](), [Database](), [Docker]() Tags: [alter](), [avoid](), [database](), [docker](), [downtime](), [migrations](), [mysql]() ### AI overview This article describes migrating 17 million rows in MySQL without downtime while a table continues receiving reads and writes. It explains using Large Hadron Migrator to copy the table and track new writes with triggers, adjusting migration speed to avoid write timeouts, and running the process in a Docker container. The migration ultimately took more than 20 hours, and cleanup was manual if problems occurred. ### Source excerpt MySQL Migrations Migrating 17M rows with no downtime - Isotope Scenario: 17M rows in a table, constant writes/reads, slave/master setup. If you try to run an ALTER TABLE (especially on older versions of MySQL), most operations will lock the table for writes for the entire run-time of the query. To avoid this, we need to incrementally update the table, incorporating write changes as they arrive. Our solution was to use the Large Hadron Migrator, built and open-sourced by SoundCloud. ## Nagios - downtime on host/service from command line with curl DevFeed: [Nagios - downtime on host/service from command line with curl]() Original publisher: [Read original article]() Author: Tom Published: 2016-01-11T00:00:00Z Content type: tutorial Language: en Sources: [Tomasz Gągor]() Topics: [Deployment](), [Bash](), [cURL](), [Command-line interface](), [Script](), [Code](), [DateTime]() Tags: [bash](), [code](), [command-line](), [configuration](), [curl](), [deployment](), [downtime](), [http](), [linux](), [monitoring](), [script](), [timezone]() ### AI overview This tutorial presents a Bash script that uses curl to schedule downtime for a Nagios host or service during deployments or other heavy tasks. It explains parameters for targeting a service or an entire host, authentication, and adapting date formats for Nagios and timezone configurations. ### Source excerpt Sometimes deployment process or other heavy task may cause some Nagios checks to rise below normal levels and bother admin1. If this is expected and you want to add downtime on host/service during this task you may use this script: #!/bin/bash function die { echo $1; exit 1; } if [[ $# -eq 0 ]] ; then die "Give hostname and time in minutes as parameter!" fi if [[ $# -eq 1 ]] ; then MINUTES=15 else MINUTES=$2 fi HOST=$1 NAGURL=http://nagios.example.com/nagios/cgi-bin/cmd.cgi USER=nagiosuser PASS=nagiospassword SERVICENAME=someservice COMMENT="Deploying new code" export MINUTES echo "Scheduling downtime on $HOST for $MINUTES minutes..." # The following is urlencoded already STARTDATE=`date "+%d-%m-%Y %H:%M:%S"` # This gives us the date/time X minutes from now ENDDATE=`date "+%d-%m-%Y %H:%M:%S" -d "$MINUTES min"` curl --silent --show-error \ --data cmd_typ=56 \ --data cmd_mod=2 \ --data host=$HOST \ --data-urlencode "service=$SERVICENAME" \ --data-urlencode "com_data=$COMMENT" \ --data trigger=0 \ --data-urlencode "start_time=$STARTDATE" \ --data-urlencode "end_time=$ENDDATE" \ --data fixed=1 \ --data hours=2 \ --data minutes=0 \ --data btnSubmit=Commit \ --insecure \ $NAGURL -u "$USER:$PASS"| grep -q "Your command request was successfully submitted to Nagios for processing." || die "Failed to con tact nagios"; echo Scheduled downtime on nagios from $STARTDATE to $ENDDATE Threat this script as template with some tips: ## Announcement: Server downtime on 28th December DevFeed: [Announcement: Server downtime on 28th December]() Original publisher: [Read original article]() Published: 2013-12-16T00:00:00Z Content type: news Language: en Sources: [Front Page on ReactOS Website]() Topics: [Availability](), [upgrade](), [Server](), [Website]() Tags: [announcement](), [availability](), [bots](), [builds](), [downtime](), [free](), [infrastructure](), [open-source](), [os](), [react](), [reactos](), [server](), [services](), [win32](), [winapi]() ### AI overview ReactOS's Infrastructure Team announced a server upgrade scheduled for 28 December to address website availability problems. The upgrade will cause downtime for several public services, while SVN and build and test bots will remain available. ### Source excerpt In order to fix the availability problems of our Website, the ReactOS Infrastructure Team will upgrade another Server on 28th December. This will result in a downtime for the following public services: www.reactos.org Websitejira.reactos.org Bugtrackerreactos.reactos.org Test ServerSVN will continue to function, as well as our Build and Test bots. The latest ReactOS builds can be downloaded at iso.reactos.org in the meantime. After the upgrade, you should be able to enjoy a higher availability and faster response times when using the ReactOS Online Services. ## Live restarts of a supervised unicorn process DevFeed: [Live restarts of a supervised unicorn process]() Original publisher: [Read original article]() Author: Sam Saffron Published: 2013-11-13T04:59:56Z Content type: tutorial Language: en Sources: [Sam Saffron]() Topics: [Scripting, bash](), [Script](), [Rails](), [Processes](), [HTTP]() Tags: [bash](), [docker](), [downtime](), [rails](), [script](), [uptime](), [zero]() ### AI overview The article describes using a Bash script as a stable-pid proxy and mini-supervisor for Unicorn to coordinate live restarts under supervisors such as runit. It also presents a throttled HTTP request script for measuring downtime during restarts and reports zero downtime with the proposed method. ### Source excerpt We have all seen this dreaded screen before. In the Rails case this usually happens during application restarts. While Discourse is rapidly evolving we are heavily encouraging users to upgrade frequently, even weekly. If your site is regularly erroring out, users very quickly lose confidence. In the ideal case you want zero downtime deploys. This feature heavily encourages users to deploy more rapidly. Unicorn has built-in support for live restarts, however getting this to play well with a supervisor like say runit is not easy. Underlying pids are changing and stuff gets complicated fast. To tackle this I decided to create a simple bash script that acts as a mini-supervisor for unicorn. However, before any of this I needed some sane way of measuring how well I did. ###Measuring uptime during a live restart Traditionally you would use apache bench for quick and dirty testing, however it did not fare well for me. Unfortunately ab has no way of "throttling" the amount of requests it sends out. To measure uptime we need to perform a request to the site every N milliseconds. I ended up knocking up a quick and dirty apache bench clone that allows me to trickle through requests: require "optparse" require "uri" require "net/http" duration = 10 per_second = 10 opts = OptionParser.new do |opts| opts.banner = "Usage: bench_web [options] url" opts.on("-t", "--time TIME", OptionParser::DecimalInteger, "Duration to run the test in seconds (default 10)") do |t| duration = t end opts.on("-p", "--per-second REQUESTS", OptionParser::DecimalInteger, "Max number of requests per second (default 10)") do |t| per_second = t.to_f end end opts.parse! if ARGV.length != 1 puts opts.banner puts exit(1) end uri = begin URI(ARGV[0]) rescue puts opt.banner puts puts "Invalid URL" puts exit(1) end GC.disable finish_time = Time.now + duration results = [] while (start=Time.now) < finish_time res = Net::HTTP.get_response(uri) req_duration = Time.now - start results << {duration: req_duration, code: [Next page]()