Using docker.io for registryOverwrite

[heilmela]

Since we regularly hit issues with registry.k8s.io i thought about using overwriteRegistry to migrate to our private mirror on docker.io.

However on dockerhub we cant use namespacing for repos such that e.g. registry.k8s.io/coredns/coredns or quay.io/calico/cni do not work. Is it possible to use dockerhub somehow ?

Btw i tried using the hack/image-loader and the script fails e.g. on quay.io/calico/cni since it tries to namespace the image on docker.io as noted above. Im able to work around this issue by modifying the cut stmmt when retagging images, but then the kk1 docs mention overwriteRegistry does keep paths like /calico/cni which woulnt work.

[xmudrii]

@heilmela The way this feature works is that only the domain part is overwritten, e.g. docker.io. That said, this is mostly like not going to work with Docker Hub. This feature was mainly made to be used with image caching solutions such as Docker Registry and Harbor. Maybe you might have more luck with Mirror Registries feature, but I personally never tried it with Docker Hub.

[embik]

we regularly hit issues with registry.k8s.io

Can you maybe let us know what issues you are facing with this registry? Is this something that can be reported to Kubernetes to improve their infrastructure?

[heilmela]

We are mainly hosting clusters on Hetzner Cloud and have seemingly random 403 Forbidden responses from different registries, including gitlab and registry.k8s.io. Sometimes its a rate limit but there is an ongoing issue with Hetzner IPs getting blocked by google. I verified this by curling https://gcping.com which rejects on affected nodes. In the past we just migrated these nodes but this effort gets costly over time. See this related issue gitlab. I didnt verify this for registry.k8s.io but it the DNS resolves on a google ip, so maybe this is the same issue here.

Some kk1 addons like hetzner csi have images e.g. livenessprobes with a Pull Policy of always, which in turn prevents the pod from starting on affected nodes even when the image is present.

[embik]

Understood. I'm not sure there's much to do here to be honest. Maybe adjusting the image pull policies. As @xmudrii pointed out, Docker Hub might not be flexible enough to be used as a mirror, given that you need nested structures to avoid naming collisions. I wonder if quay.io might be of more help to you.

[heilmela]

Yes another option would be AWS ECR which allows to work around namespaces with repository names including the "domain path". However put simply, we already host a lot on dockerhub and have corresponding licenses, thus i would prefer on staying there if it works somehow. Im gonna check mirror registries first and keep the thread updated if it works.