# Redis latency spikes and the Linux kernel: a few more details

DevFeed: [Redis latency spikes and the Linux kernel: a few more details](<https://devfeed.tech/articles/redis-latency-spikes-and-the-linux-kernel-a-few-more-details-20675.md>)

Original publisher: [Read original article](<http://antirez.com/news/84>)

Published: 2014-11-03T15:58:19Z

Content type: article

Language: en

Sources: [Antirez](<https://devfeed.tech/sources/antirez.md>)

Topics: [Redis](<https://devfeed.tech/topics/redis.md>), [Latency](<https://devfeed.tech/topics/latency.md>), [Linux](<https://devfeed.tech/topics/linux.md>), [Kernel](<https://devfeed.tech/topics/kernel.md>), [Amazon EC2](<https://devfeed.tech/topics/amazon-ec2.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [blocking](<https://devfeed.tech/tags/blocking.md>), [ec2](<https://devfeed.tech/tags/ec2.md>), [fork](<https://devfeed.tech/tags/fork.md>), [latency](<https://devfeed.tech/tags/latency.md>), [linux](<https://devfeed.tech/tags/linux.md>), [linux-kernel](<https://devfeed.tech/tags/linux-kernel.md>), [memory](<https://devfeed.tech/tags/memory.md>), [process](<https://devfeed.tech/tags/process.md>), [redis](<https://devfeed.tech/tags/redis.md>), [signal](<https://devfeed.tech/tags/signal.md>)

## AI overview

The article investigates roughly 300-millisecond Redis latency spikes during BGSAVE on EC2 instances. Tests and stack traces indicated that the delays occurred after fork, near memory operations in the parent process, leading to a preliminary theory involving Linux copy-on-write behavior and the fork implementation.

## Source excerpt

Today I was testing Redis latency using m3.medium EC2 instances. I was able to replicate the usual latency spikes during BGSAVE, when the process forks, and the child starts saving the dataset on disk. However something was not as expected. The spike did not happened because of disk I/O, nor during the fork() call itself. The test was performed with a 1GB of data in memory, with 150k writes per second originating from a different EC2 instance, targeting 5 million keys (evenly distributed). The pipeline was set to 4 commands. This translates to the following command line of redis-benchmark: ./redis-benchmark -P 4 -t set -r 5000000 -n 1000000000 Every time BGSAVE was triggered, I could see ~300 milliseconds latency spikes of unknown origin, since fork was taking 6 milliseconds. Fortunately Redis has a software watchdog feature, that is able to produce a stack trace of the process during a latency event. It's quite a simple trick but works great: we setup a SIGALRM to be delivered by the kernel. Each time the serverCron() function is called, the scheduled signal is cleared, so actually Redis never receives it if the control returns fast enough to the Redis process. If instead there is a blocking condition, the signal is delivered by the kernel, and the signal handler prints the stack trace. Instead of getting stack traces with the fork call, the process was always blocked near MOV* operations happening in the context of the parent process just after the fork. I started to develop the theory that Linux was "lazy forking" in some way, and the actual heavy stuff was happening later when memory was accessed and pages had to be copy-on-write-ed. Next step was to read the fork() implementation of the Linux kernel. What the system call does is indeed to copy all the mapped regions (vm_area_struct structures). However a traditional implementation would also duplicate the PTEs at this point, and this was traditionally performed by copy_page_range(). However something changed... a