# Varnish to Redis Migration

DevFeed: [Varnish to Redis Migration](<https://devfeed.tech/articles/varnish-to-redis-migration-26226.md>)

Original publisher: [Read original article](<https://medium.com/groupon-eng/varnish-to-redis-migration-887ad6d805d2?source=rss----5c13a88f9872---4>)

Author: Ravikumar

Published: 2022-03-22T15:39:20Z

Content type: article

Language: en

Sources: [Groupon Engineering -- Medium](<https://devfeed.tech/sources/groupon-engineering-medium.md>)

Topics: [migration](<https://devfeed.tech/topics/migration.md>), [Caching](<https://devfeed.tech/topics/caching.md>), [Amazon EKS](<https://devfeed.tech/topics/amazon-eks.md>), [Redis](<https://devfeed.tech/topics/redis.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>)

Tags: [aws](<https://devfeed.tech/tags/aws.md>), [cache-invalidation](<https://devfeed.tech/tags/cache-invalidation.md>), [caching](<https://devfeed.tech/tags/caching.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [distributed-transaction](<https://devfeed.tech/tags/distributed-transaction.md>), [haproxy](<https://devfeed.tech/tags/haproxy.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [migration](<https://devfeed.tech/tags/migration.md>), [nginx](<https://devfeed.tech/tags/nginx.md>), [proxy](<https://devfeed.tech/tags/proxy.md>), [redis](<https://devfeed.tech/tags/redis.md>), [varnish](<https://devfeed.tech/tags/varnish.md>)

## AI overview

Groupon describes migrating its Varnish caching infrastructure to Redis while moving application workloads to AWS EKS. The article focuses on a high-traffic place read service, the limitations of TTL-based cache invalidation, and the planned shift to event-based expiration.

## Source excerpt

At Groupon, we are in the process of migrating our app workloads to AWS EKS. As part of the migration, we are also in the process of re-architecting our services to make them cloud & Kubernetes optimised. In this article, we will focus on the impact of cloud migration on Varnish caching at Groupon. For the uninitiated, Varnish is a high-performance reverse caching proxy. For simplicity, you can think of it as a general proxy such as HAProxy or Nginx with caching support so that it doesn't have to go to a backend service every time it receives a request. Internally, Varnish cache uses pthreads heavily(to the tune of 1000s) to serve the requests. In contrast, Redis is single-threaded. Varnish at Groupon & Place Read Service We have been using Varnish for many of our high-traffic services. These read-heavy services get traffic to the tune of millions of RPM. One such service is our place read service which is called on each & every deal page visit; it will be the focus of this article. Place read service stores all the locations a deal can be redeemed. Ex: Redemption location details of the deal https://www.groupon.com/deals/parent-big-littles-3 is highlighted below. Traffic pattern of place read service The service is called more than a billion times a day with peak traffic of ~2 million requests per minute. Current architecture (simplified) The traffic is served by a cluster of 4 Varnish servers (40GB RAM). The current Varnish hit rate is 95%. This is admittedly a low hit rate for a read-heavy service, the main reason being our TTL-based cache invalidation logic. Varnish cache invalidation We are using TTL-based cache invalidation. This usually leads to a problem where Varnish starts bombarding the backend service when most of the keys expire at the same time. Request coalescing is not very helpful when the unique keys count is in millions (which is the case with our service). As a workaround, we use randomised TTLs (between 30 to 60 mins) so that all the keys don't