# Redis latency spikes and the 99th percentile

DevFeed: [Redis latency spikes and the 99th percentile](<https://devfeed.tech/articles/redis-latency-spikes-and-the-99th-percentile-20674.md>)

Original publisher: [Read original article](<http://antirez.com/news/83>)

Published: 2014-10-30T13:28:42Z

Content type: opinion

Language: en

Sources: [Antirez](<https://devfeed.tech/sources/antirez.md>)

Topics: [Latency](<https://devfeed.tech/topics/latency.md>), [Redis](<https://devfeed.tech/topics/redis.md>), [Amazon EC2](<https://devfeed.tech/topics/amazon-ec2.md>), [Processes](<https://devfeed.tech/topics/processes.md>), [servers](<https://devfeed.tech/topics/servers.md>)

Tags: [ec2](<https://devfeed.tech/tags/ec2.md>), [fork](<https://devfeed.tech/tags/fork.md>), [http](<https://devfeed.tech/tags/http.md>), [latency](<https://devfeed.tech/tags/latency.md>), [process](<https://devfeed.tech/tags/process.md>), [redis](<https://devfeed.tech/tags/redis.md>), [server](<https://devfeed.tech/tags/server.md>)

## AI overview

The article examines Redis latency spikes caused by fork() during disk persistence, particularly on Xen-based EC2 and other virtual servers. It explains how occasional one-second spikes can distort 99th-percentile measurements and differ from evenly distributed latency spikes that affect page views.

## Source excerpt

One interesting thing about the Stripe blog post about Redis is that they included latency graphs obtained during their tests. In order to persist on disk Redis requires to call the fork() system call. Usually forking using physical servers, and most hypervisors, is fast even with big processes. However Xen is slow to fork, so with certain EC2 instance types (and other virtual servers providers as well), it is possible to have serious latency spikes every time the parent process forks in order to persist on disk. The Stripe graph is pretty clear in this regard. img://antirez.com/misc/stripe-latency.png As you can guess, if you perform a latency test during the fork, all the requests crossing the moment the parent process forks will be delayed up to one second (taking as example the graph above, not sure about what was the process size nor the EC2 instance). This will produce a number of samples with high latency, and will affect the 99th percentile result. To change instance type, configuration, setup, or whatever in order to improve this behavior is a good idea, and there are use cases where even a single request having a too high latency is unacceptable. However apparently it is not obvious how latency spikes of 1 second every 30 minutes (or more, if you use AOF with the right rewrite triggers) is very different from latency spikes which are evenly distributed in the set of requests. With evenly distributed spikes, if the generation of a page needs to perform a number of requests to a Redis server in order to create the output, it is very likely that a page view will incur in the latency penalty: this impacts the quality of service in a great way potentially, check this link: http://latencytipoftheday.blogspot.it/2014/06/latencytipoftheday-most-page-loads.html. However 1 second of latency every 30 minutes run is a completely different thing. For once, the percentile with good latency gets better *as the number of requests increase*, since the more the requests are