# Docker Hub pull rate limits hit Rahti

DevFeed: [Docker Hub pull rate limits hit Rahti](<https://devfeed.tech/articles/docker-hub-pull-rate-limits-hit-rahti-19775.md>)

Original publisher: [Read original article](<https://cloud.blog.csc.fi/2021/03/docker-hub-pull-rate-limits-hit-rahti.html>)

Author: Unknown (noreply@blogger.com)

Published: 2021-03-04T08:59:00Z

Content type: article

Language: en

Sources: [CSC - IT Center For Science - Cloud Team](<https://devfeed.tech/sources/csc-it-center-for-science-cloud-team.md>)

Topics: [Docker Hub](<https://devfeed.tech/topics/docker-hub.md>), [incident](<https://devfeed.tech/topics/incident.md>), [Availability](<https://devfeed.tech/topics/availability.md>), [Caching](<https://devfeed.tech/topics/caching.md>), [Back end](<https://devfeed.tech/topics/backend.md>)

Tags: [availability](<https://devfeed.tech/tags/availability.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [docker-container](<https://devfeed.tech/tags/docker-container.md>), [docker-hub](<https://devfeed.tech/tags/docker-hub.md>), [incident](<https://devfeed.tech/tags/incident.md>), [rate-limiting](<https://devfeed.tech/tags/rate-limiting.md>), [registry](<https://devfeed.tech/tags/registry.md>)

## AI overview

This technical article describes how Docker Hub pull rate limits affected Rahti deployments and builds, and how administrators responded by recommending alternative repositories and caching or migrating images to Rahti's registry. The impact was compounded by availability problems in Rahti's own registry during a storage incident.

## Source excerpt

November 2020 was an eventful month for Rahti. Our backend storage had incident that affected our production clusters image repository for many weeks. That event is well described in Allas downtime November 2020 - technical deep-dive. Just before our storage incident, 2nd of November, Docker applied "rate limiting for Docker container pulls for some users". When we tried to prepare for this in advance, it was very hard to figure out what the exact effect would be for Rahti. It took longer to resolve than we anticipated. Limits beganWe were relieved that the first day after pull rates were enforced was peaceful. Rahti was still up and running as it should. Users didn't complain about failing deployment in large amounts. Admins survived another upstream change. Eventually we received few tickets on failing builds. Our first approach was to instruct people to change their images away from Docker Hub to alternative repositories which do not have rate limiting enabled. Some users were able to migrate their images but this was not feasible solution for everyone. Still, it was a good opportunity to remind users to check what images they use. The error looks like this: Pulling image "docker.io/centos/python-38-centos7@sha256:da83741689a8d7fe1548fefe7e001c45bcc56a08bc03fd3b29a5636163ca0353" ... pulling image error : toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit Cumulative issues cause exponential griefWe gave our users instructions how to move their dependencies from public Docker Hub to Rahti image registry, How to manually cache images in Rahti's registry. This was harder in practice, as we did have issues with our own registry due to our storage incident. There were days when users suffered from both Docker Hub and Rahti internal registry availability. Those days were hard for our Rahti admins, who struggled to meet customer expectations of highly available ser