2d ago

Docker Hub limiting unauthenticated users to 10 pulls per hour

Learn about usage and limits for Docker Hub.

AFAIK every NAS just uses unauthenticated connections to pull containers, I'm not sure how many actually allow you to log in even (raising the limit to a whopping 40 per hour).

So hopefully systems like /r/unRAID handle the throttling gracefully when clicking "update all".

Anyone have ideas on how to set up a local docker hub proxy to keep the most common containers on-site instead of hitting docker hub every time?

Technology @lemmy.world

nahostdeutschland @feddit.org

2d ago

Docker limits unauthenticated pulls to 10/hour/IP from Docker Hub, from March 1

docs.docker.com /docker-hub/usage/

68 9

46 comments

and now I don't sound so fucking stupid for setting up local image caches on my self-hosted gitlab server.

Forgejo gives you a registry built-in.
Also is it just me or does the docker hub logo look like it's giving us the middle finger?

Fortunately linuxserver's main hosting is no longer dockerhub.
- Would you be able to share more info? I remember reading their issues with docker, but I don't recall reading about whether or what they switched to. What is it now?
  
  They run their own registry at lscr.io. You can essentially prefix all your existing linuxserver image names with lscr.io/ to pull them from there instead.

How long since getting an oracle CEO did this take?
- Did they really? Oh my god please tell me your joking, that a company as modern as docker got a freaking oracle CEO. They pulled a Jack Barker. Did he bring his conjoined triangles of success?
  
  A "jack barker" 🤣

Use a service that's not Docker hub

Huh. I was just considering establishing a caching registry for other reasons. Ferb, I know what we’re going to do today!
- Do you have a good resource for how one can go about this?
  
  You can host your own with harbor, and set up replication per repo (pull upstream tags) If you need a commercial product/support you can use MSR v4.
  Harbor can install on any K8s cluster using helm, with just a couple of dependencies (cert-manager, postgres op, redis-op) Replication stuff you can easily add.
  I have some no-warranty terraform I could share if there is some interest.
  
  I’ve only done my “is it even possible” research so far, but these look promising:
  https://medium.com/@amandubey_6607/docker-registry-caching-a2dfefecfff5
  https://github.com/obeone/multi-registry-cache
- Same here. I've been building a bootstrap script, and each time I test it, it tears down the whole cluster and starts from scratch, pulling all of the images again. Every time I hit the Docker pull limit after 10 - 12 hours of work, I treat that as my "that's enough work for today" signal. I'm going to need to set up a caching system ASAP or the hours I work on this project are about to suddenly get a lot shorter.

Instead of using a sort of Docker Hub proxy, you can also use GitHub's repository or Quay. If the project allows it, you can easily switch to these alternatives. Alternatively, you can build the Docker image yourself from the source. It's usually not a difficult process, as most of it is automated. Or what I personally would probably do is just update the image a day later if I hit the limit.
- You can also host your own with harbor (or MSR v4 if you want a commercial product.) You can set them up to replicate upstream.

I'm quite new to docker for NAS stuff - how many pulls would the average person do? like, i don't think i even have 10 containers 🤨
- I'm running ~30 containers, but they don't typically all get new updates at the same time.
  Updates are grabbed nightly, and I think the most I've seen update at once is like 6 containers.
  Could be a problem for setting up a new system, or experimenting with new toys.
  
  The problem is that the main container can (and usually does) rely on other layers, and you may need to pull updates for those too. Updating one app can take 5-10 individual pulls.

https://distribution.github.io/distribution/
is an opensource implementation of a registry.
you could also self host something like gitlab, which bundles this or sonatype nexus which can serve as a repository for several kinds of artifacts including container images.
- Gitea and therefore Forgejo also have container registry functionality, I use that for private builds.
  
  Codeberg as woodpecker CI
  
  Jumping on the forgejo love train
  
  oh, thats good to know, forgejo seems way nicer for self hosting than the limited gitlab open source core.

Well shit, I still rely on Docker Hub even for automated pulls so this is just great. I guess i'm going back to managing VMs with OpenTofu and package managers.
What are our alternatives if we use Podman or K8s?
- The issue isn't Docker vs Podman vs k8s ~~vs LXC~~ vs others. They all use OCI images to create your container/pod/etc. This new limit impacts all containerization solutions, not just Docker. EDIT: removed LXC as it does not support OCI
  Instead, the issue is Docker Hub vs Quay vs GHCR vs others. It's about where the OCI images are stored and pulled from. If the project maintainer hosts the OCI images on Docker Hub, then you will be impacted by this regardless of how you use the OCI images.
  Some options include:
  For projects that do not store images on Docker Hub, continue using the images as normal
  Become a paid Docker member to avoid this limit
  When a project uses multiple container registries, use one that is not Docker Hub
  For projects that have community or 3rd party maintained images on registries other than Docker Hub, use the community or 3rd party maintained images
  For projects that are open source and/or have instructions on building OCI images, build the images locally and bypass the need for a container registry
  For projects you control, store your images on other image registries instead of (or in addition to) Docker Hub
  Use an image tag that is updated less frequently
  Rotate the order of pulled images from Docker Hub so that each image has an opportunity to update
  Pull images from Docker Hub less frequently
  For images that are used by multiple users/machine under your supervision, create an image cache or image registry of images that will be used by your users/machines to mitigate the number of pulls from Docker Hub
  Encourage project maintainers to store images on image registries other than Docker Hub (or at least provide additional options beyond Docker Hub)
  Do not use OCI images and either use VM or bare metal installations
  Use alternative software solutions that store images on registries other than Docker Hub
  
  Lxc doesn't use oci images? I always end up using docker in lxc when dockeris the only option (which I have not figured how to makw work on my airgapped side

local docker hub proxy
Do you mean a Docker container registry? If so, here are a couple options:
Use the official Docker registry: https://www.docker.com/blog/how-to-use-your-own-registry-2/
Self-host forgejo or gitea and use the included package registry, which is automatically enabled. Details: https://forgejo.org/docs/latest/user/packages/
- I think they mean pull-through cache. https://shipyard.build/blog/how-to-docker-registry-pull-through-cache/

Is there a project that acts like a registry? It can proxy the request with TTL, and you can push images to it too?
- Almost all of them. Forgejo handles containers already for example
  
  How? I was looking for this (although not very thoroughly)
  [Edit] found it https://forgejo.org/docs/v1.21/user/packages/container/
  
  Pull through Cache / proxy is what you're looking for.
- Artifactory is mandatory in some industries because it will keep all the versions of the images forever so that you can build your projects reliably without an internet connction.
- I think most self-hosted Git+CI/CD platforms have container registry as a feature, but I'm not aware of a service that is just a standalone registry.
  
  It's easy to oversee because of the generic name, but this is pretty much that: https://hub.docker.com/_/registry
  Edit: forgot there's jfrog artifactory as well

If only they used a distributed protocol like ipfs, we wouldn't be in this situation

46 comments