How to deal with Docker Hub rate limit on AWS

Emmanuel Gautier / November 11, 2021

4 min read

Since 2020, Docker Hub service has been rate limited to only 100, 200 container image pull requests per six hours. This limit is fair enough for personal or small team projects but can be a real limit if you have CI jobs or infrastructure deployment process pulling images from Docker hub registry. This article will help you to deal with this limitation on AWS.

During your AWS Codebuild run or during an ECS task definition instantiation, AWS Services will try by default to pull images from Docker Hub registry. At the moment, the Docker Hub service rate limit can make the process fail. So we need to deal with this limitation. Many ways to do this are available.

What is Docker Hub limit?

Docker Hub put a limit on the number of container image requests an account can pull. This limit is set to 100, 200 requests per six hours and detected per user which is mostly IP address when the request is not authenticated.

If you reach this limit, the request failed and the image can't be pulled making your docker command fail. This error can show up with this error message:

ERROR: toomanyrequests: Too Many Requests.

or

You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limits.

If you want to know more about the Docker Hub limit, please visit this article

Upgrade your Docker Hub account

A simple first option is to authenticate your request and upgrade your Docker Hub account. This will allow you to pull more images. To make this upgrade, you can have a look at Docker Hub pricing page.

This option is the easiest way, and surely the fastest way to deal with the Docker Hub limit. But other options are available using more AWS Services.

Limit pull requests during the build with cache layers

If you are building a new image from another image, you can use the --cache-from allowing you to reuse layers from the previous image built.

For example, if you make a new build of an image myname/myapp, you can use the following command:

docker build --cache-from myname/myapp .

Use AWS Public Images Registry

Since 2020, AWS had a Public Images Registry (AWS ECR) service. This service is a public registry that allows you to store, share and pull images outside of Docker Hub registry. This service is available on AWS with a free tier.

The issue is that this public registry does not have as many images choices Docker Hub has. But it is a good option to reduce the number of images pulled from Docker Hub.

Here is the link to the AWS Public ECR registry: https://gallery.ecr.aws/

Use AWS ECR

The last option is to use AWS ECR service. This service is a private registry that allows you to store and pull your own images privately. You can both use private images from your own AWS ECR repositories and public images from AWS Public ECR repositories.

To pull and push images from AWS ECR, you need first to create a repository on AWS ECR. This repository will be used to store your images.

Then you will need to retrieve an authentication token:

aws ecr get-login-password --region <region> | docker login --username AWS --password-stdin <account>.dkr.ecr.<region>.amazonaws.com

To be able to push your images you will need to tag the image with the repository URL:

docker tag <image>:<tag> <account>.dkr.ecr.<region>.amazonaws.com/<image>:<tag>

Now you can push your newly tagged image.

Hope this article helped you to deal with the Docker Hub rate limitations on AWS.