# A few arguments about Redis Sentinel properties and fail scenarios.

DevFeed: [A few arguments about Redis Sentinel properties and fail scenarios.](<https://devfeed.tech/articles/a-few-arguments-about-redis-sentinel-properties-and-fail-scenarios-20671.md>)

Original publisher: [Read original article](<http://antirez.com/news/80>)

Published: 2014-10-21T15:18:10Z

Content type: opinion

Language: en

Sources: [Antirez](<https://devfeed.tech/sources/antirez.md>)

Topics: [Redis](<https://devfeed.tech/topics/redis.md>), [distributed-systems](<https://devfeed.tech/topics/distributed-systems.md>), [systems](<https://devfeed.tech/topics/systems.md>), [incident](<https://devfeed.tech/topics/incident.md>)

Tags: [backups](<https://devfeed.tech/tags/backups.md>), [crash](<https://devfeed.tech/tags/crash.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [distributed-systems](<https://devfeed.tech/tags/distributed-systems.md>), [incident](<https://devfeed.tech/tags/incident.md>), [persistence](<https://devfeed.tech/tags/persistence.md>), [processes](<https://devfeed.tech/tags/processes.md>), [redis](<https://devfeed.tech/tags/redis.md>)

## AI overview

The article examines a Redis Sentinel incident initially attributed to a serious failover bug. It explains that disk persistence had been disabled, causing the master to restart with an empty dataset, and discusses process restarts, state loss, crash recovery, and a separate Sentinel behavior the author believes could be improved.

## Source excerpt

Yesterday distributed systems expert Aphyr, posted a tweet about a Redis Sentinel issue experienced by an unknown company (that wishes to remain anonymous): "OH on Redis Sentinel "They kill -9'd the master, which caused a split brain..." "then the old master popped up with no data and replicated the lack of data to all the other nodes. Literally had to restore from backups." OMG we have some nasty bug I thought. However I tried to get more information from Kyle, and he replied that the users actually disabled disk persistence at all from the master process. Yep: the master was configured on purpose to restart with a wiped data set. Guess what? A Twitter drama immediately started. People were deeply worried for Redis users. Poor Redis users! Always in danger. However while to be very worried is a trait of sure wisdom, I want to take the other path: providing some more information. Moreover this blog post is interesting to write since actually Kyle, while reporting the problem with little context, a few tweets later was able to, IMHO, isolate what is the true aspect that I believe could be improved in Redis Sentinel, which is not related to the described incident, and is in my TODO list for a long time now. But before, let's check a bit more closely the behavior of Redis / Sentinel about the drama-incident. Welcome to the crash recovery system model === Most real world distributed systems must be designed to be resilient to the fact that processes can restart at random. Note that this is very different from the problem of being partitioned away, which is, the inability to exchange messages with other processes. It is, instead, a matter of losing state. To be more accurate about this problem, we could say that if a distributed algorithm is designed so that a process must guarantee to preserve the state after a restart, and fails to do this, it is technically experiencing a bizantine failure: the state is corrupted, and the process is no longer reliable. Now in a distri