How to deal with Docker Hub rate limit on AWS

Since 2020, Docker Hub service has been rate limited to only 100, 200 container image pull requests per six hours. This limit is fair enough for personal or small team projects but can be a real limit if you have CI jobs or infrastructure deployment process pulling images from Docker hub registry. This article will help you to deal with this limitation on AWS.

During your AWS Codebuild run or an ECS task instantiation, AWS Services will try by default to pull images from Docker Hub registry. At the moment, the Docker Hub service rate limit can make the process fail. So we need to deal with this limitation. Many ways to do this are available.

What is Docker Hub limit?

Docker Hub put a limit on the number of container image requests an account can pull. This limit is set to 100, 200 requests per six hours and detected per user which is mostly IP address when the request is not authenticated.

If you reach this limit, the request failed and the image can't be pulled making your docker command fail. This error can show up with this error message:

ERROR: toomanyrequests: Too Many Requests.

or

You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limits.

If you want to know more about the Docker Hub limit, please visit this article

Upgrade your Docker Hub account

A simple first option is to authenticate your request and upgrade your Docker Hub account. This will allow you to pull more images. To make this upgrade, you can have a look at Docker Hub pricing page.

This option is the easiest way, and surely the fastest way to deal with the Docker Hub limit. But other options are available using more AWS Services.

Limit pull requests during the build with cache layers

If you are building a new image from another image, you can use the --cache-from allowing you to reuse layers from the previous image built.

For example, if you make a new build of an image myname/myapp, you can use the following command:

docker build --cache-from myname/myapp .

Use AWS Public Images Registry

Since 2020, AWS had a Public Images Registry (AWS ECR) service. This service is a public registry that allows you to store, share and pull images outside of Docker Hub registry. This service is available on AWS with a free tier.

The issue is that this public registry does not have as many images choices Docker Hub has. But it is a good option to reduce the number of images pulled from Docker Hub.

Here is the link to the AWS Public ECR registry: https://gallery.ecr.aws/

Use Amazon Elastic Container Registry

The last option is to use AWS ECR service. This service is a private registry that allows you to store and pull your images privately. You can both use private images from your AWS ECR repositories and public images repositories hosted elsewhere.

Since end of November 2021, AWS announce the new Pull Through Cache feature. You can now use this feature to pull images from Docker Hub through ECR with a cache layer. This cache layer allows to reduce the number of images pulled from Docker Hub and exceed the rate limit.

To pull and push images from AWS ECR, you need first to create a repository namespace on AWS ECR and configure a pull-through cache rule to the Docker Hub destination.

Then you will need to retrieve an authentication token:

aws ecr get-login-password --region <region> | docker login --username AWS --password-stdin <account>.dkr.ecr.<region>.amazonaws.com/docker-hub

You can now pull images from Docker Hub through AWS ECR:

docker pull <account>.dkr.ecr.<region>.amazonaws.com/docker-hub/<image>:<tag>

Hope this article helped you to deal with the Docker Hub rate limitations on AWS.