How to setup reigistry-mirror in Kind for "failed to pull image" error

[xushijie]

I follow this page: https://docs.ray.io/en/latest/cluster/kubernetes/getting-started/raycluster-quick-start.html#kuberay-raycluster-quickstart and install ray-clustte using command helm install raycluster kuberay/ray-cluster --version 1.2.2.

After execution, the kind can not be uanble to pull images:
`

-> kubectl get pods

-> kubectl describe pod raycluster-kuberay-head-krpmp

....
Events:
Type Reason Age From Message
Warning Failed 28m (x42 over 3h21m) kubelet (combined from similar events): Failed to pull image "rayproject/ray:2.9.0": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/rayproject/ray:2.9.0": failed to resolve reference "docker.io/rayproject/ray:2.9.0": failed to do request: Head "https://registry-1.docker.io/v2/rayproject/ray/manifests/2.9.0": dial tcp 157.240.2.14:443: i/o timeout
Normal Pulling 23m (x43 over 3h58m) kubelet Pulling image "rayproject/ray:2.9.0"
Normal BackOff 3m2s (x938 over 3h57m) kubelet Back-off pulling image "rayproject/ray:2.9.0"
`

My question is how can I to config registry mirror when using Kind? The document says KIND uses docker container, while the docker do can pull these images with my own register-image:
`

-> docker  info
..............
Registry Mirrors:
  https://docker.registry.cyou/
  https://docker-cf.registry.cyou/
  https://proxy.1panel.live/
  https://docker.1panel.top/
  https://docker.m.daocloud.io/
  https://docker.1ms.run/
  https://5wxalzzb.mirror.aliyuncs.com/
 Live Restore Enabled: false
-> docker images |grep ray 
rayproject/ray                                             2.34.0                    36adde223ebb   5 months ago    2.22GB
rayproject/ray                                             2.9.0                     ee1c865e0257   12 months ago   2.2GB

`

additional information is that I can also pull this image via crictl as I configured registery-mirror for containerd:
`

base) ➜  sudo crictl pull docker.io/rayproject/ray:2.9.0

Image is up to date for sha256:ee1c865e02572031f5c68be4184f21d929d5cd233289c83b2a10b9ce455eb75f
(base) ➜  sudo crictl images 
IMAGE                                                             TAG                 IMAGE ID            SIZE
docker.io/rayproject/ray                                          2.9.0               ee1c865e02572       747MB
registry.aliyuncs.com/google_containers/coredns                   v1.10.1             ead0a4a53df89       16.2MB
registry.aliyuncs.com/google_containers/etcd                      3.5.7-0             86b6af7dd652c       102MB
registry.aliyuncs.com/google_containers/kube-apiserver            v1.27.1             6f6e73fa8162b       33.4MB
registry.aliyuncs.com/google_containers/kube-controller-manager   v1.27.1             c6b5118178229       31MB
registry.aliyuncs.com/google_containers/kube-proxy                v1.27.1             fbe39e5d66b6a       23.9MB
registry.aliyuncs.com/google_containers/kube-scheduler            v1.27.1             6468fa8f98696       18.2MB
registry.aliyuncs.com/google_containers/pause                     3.9                 e6f1816883972       322kB

`

[stmcginnis]

Kind uses containerd and not Docker, so that will be the relevant part to look at. You can patch the containerd config inside the node that gets deployed by providing your custom mirror configuration under containerdConfigPatches, similar to how the docs show for setting up certificates.

You could also consider using a local registry to load your images.

[xushijie]

Kind uses containerd and not Docker, so that will be the relevant part to look at. You can patch the containerd config inside the node that gets deployed by providing your custom mirror configuration under containerdConfigPatches, similar to how the docs show for setting up certificates.

You could also consider using a local registry to load your images.

Thanks for your reply.
Any idea the this pull fail even when the crictl pulls success? I can pull it using crictl and you can see crtctl images cmd output.
Based on your explaination, the containerd is used and crictl is a interface for K8S to intereact with container.

[stmcginnis]

Any idea the this pull fail even when the crictl pulls success?

Are you running that from the host or from the cluster node? The cluster node doesn't use your host OS configuration, so to do a proper test you will need to exec in to the node container and run commands from there.

From there you can also verify your settings from containerdConfigPatches have successfully been applied to the node's configuration.

[xushijie]

Any idea the this pull fail even when the crictl pulls success?

Are you running that from the host or from the cluster node? The cluster node doesn't use your host OS configuration, so to do a proper test you will need to exec in to the node container and run commands from there.

From there you can also verify your settings from containerdConfigPatches have successfully been applied to the node's configuration.

Thanks for your suggestion. I have not tried containerdConfigPatches yet, and will try this one some times later.
In my side, it is a single node cluster, ie, all in my laptop for experiment. I finally solve it via kind load docker-image rayproject/ray:2.9.0 in your given link. The flow is:
crictl pull image --> load image to the kind cluster in the host (ie, both host and cluster are in the same laptop) -> kubectl get pods shows running pod successfully.

The 2nd step should be unnecessary here.