How to deal with Docker Hub rate limit on AWS
Emmanuel Gautier / November 11, 2021
4 min read
Since 2020, Docker Hub service has been rate limited to only 100, 200 container image pull requests per six hours. This limit is fair enough for personal or small team projects but can be a real limit if you have CI jobs or infrastructure deployment process pulling images from Docker hub registry. This article will help you to deal with this limitation on AWS.
During your AWS Codebuild run or an ECS task instantiation, AWS Services will try by default to pull images from Docker Hub registry. At the moment, the Docker Hub service rate limit can make the process fail. So we need to deal with this limitation. Many ways to do this are available.
What is Docker Hub limit?
Docker Hub put a limit on the number of container image requests an account can pull. This limit is set to 100, 200 requests per six hours and detected per user which is mostly IP address when the request is not authenticated.
If you reach this limit, the request failed and the image can't be pulled making your docker command fail. This error can show up with this error message:
ERROR: toomanyrequests: Too Many Requests.
You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limits.
If you want to know more about the Docker Hub limit, please visit this article
Upgrade your Docker Hub account
A simple first option is to authenticate your request and upgrade your Docker Hub account. This will allow you to pull more images. To make this upgrade, you can have a look at Docker Hub pricing page.
This option is the easiest way, and surely the fastest way to deal with the Docker Hub limit. But other options are available using more AWS Services.
Limit pull requests during the build with cache layers
If you are building a new image from another image, you can use the
--cache-from allowing you to reuse layers from the previous image built.
For example, if you make a new build of an image
myname/myapp, you can use the following command:
docker build --cache-from myname/myapp .
Use AWS Public Images Registry
Since 2020, AWS had a Public Images Registry (AWS ECR) service. This service is a public registry that allows you to store, share and pull images outside of Docker Hub registry. This service is available on AWS with a free tier.
The issue is that this public registry does not have as many images choices Docker Hub has. But it is a good option to reduce the number of images pulled from Docker Hub.
Here is the link to the AWS Public ECR registry: https://gallery.ecr.aws/
Use Amazon Elastic Container Registry
The last option is to use AWS ECR service. This service is a private registry that allows you to store and pull your images privately. You can both use private images from your AWS ECR repositories and public images repositories hosted elsewhere.
Since end of November 2021, AWS announce the new
Pull Through Cache feature. You can now use this feature to pull images from Docker Hub through ECR with a cache layer. This cache layer allows to reduce the number of images pulled from Docker Hub and exceed the rate limit.
To pull and push images from AWS ECR, you need first to create a repository namespace on AWS ECR and configure a pull-through cache rule to the Docker Hub destination.
Then you will need to retrieve an authentication token:
aws ecr get-login-password --region <region> | docker login --username AWS --password-stdin <account>.dkr.ecr.<region>.amazonaws.com/docker-hub
You can now pull images from Docker Hub through AWS ECR:
docker pull <account>.dkr.ecr.<region>.amazonaws.com/docker-hub/<image>:<tag>
Hope this article helped you to deal with the Docker Hub rate limitations on AWS.
If you're seeking solutions to a problem or need expert advice, I'm here to help! Don't hesitate to book a call with me for a consulting session. Let's discuss your situation and find the best solution together.
How to fix DNS Internal Resolution inside the Nginx docker container
Sometimes, errors with Docker and some containers can become cryptic when it is about networking. Today, it was Nginx that was not able to reach another container.
MySQL Docker Image for Mac ARM M1
MySQL official Docker image is not available for Apple ARM-based M1 and M2 CPUs. But there is some options to use mysql image on Mac and more globally on ARMv8 microarchitecture.
How to enable Python type checking in VSCode
Python now has support for type hints. In this article, we will see how to enable better IntelliSense and type checking analysis in VSCode.
How to manage Internationalization with NextJS SSG
Staticaly generating a website with the NextJS framework in different languages is not so obvious.
Install and configure a DNS server with Bind9 on Linux
A service DNS (Domain Name Service) allows domain name resolution to an IP Address and other resources. This service is useful for example for browsing internet websites and not have to know IPs addresses for these websites.