how-to pull local container images

[nskalis]

Hi,

I am encountering an issue when trying to pull a container image in an rke2 cluster using the following example: ghcr.io/cloudnative-pg/cloudnative-pg:1.24.1. Despite following these steps, the image does not seem to be pulled successfully:

1. Pull the image locally:
   docker pull ghcr.io/cloudnative-pg/cloudnative-pg:1.24.1

2. Save the image as a tar file:
   docker save -o cloudnative-pg-1.24.1.tar ghcr.io/cloudnative-pg/cloudnative-pg:1.24.1

3. Import the image into the container runtime:
   ctr -n k8s.io images import cloudnative-pg-1.24.1.tar

4. Tag the image for local use:
   ctr -n k8s.io images tag ghcr.io/cloudnative-pg/cloudnative-pg:1.24.1 cloudnative-pg:1.24.1

5. Remove the original image reference (optional):
   ctr -n k8s.io images rm ghcr.io/cloudnative-pg/cloudnative-pg:1.24.1

6. Verify the imported image:

   # ctr -n k8s.io images list
   REF                   TYPE                                       DIGEST                                                                  SIZE      PLATFORMS   LABELS 
   cloudnative-pg:1.24.1 application/vnd.oci.image.manifest.v1+json sha256:fd27a3305dbbe2daa62efb60e2565797811b824920ed9a634b7c9e8e9d4f716d 112.7 MiB linux/amd64 -
   

However, when I check the pod status using kubectl, the image fails to load properly, and I encounter an ErrImagePull error:

> kubectl -n cnpg-system describe pod cnpg-controller-manager-5d69854679-h8zjl
Name:             cnpg-controller-manager-5d69854679-h8zjl
Namespace:        cnpg-system
Priority:         0
Service Account:  cnpg-manager
Node:             vd013047/172.28.142.236
Start Time:       Tue, 28 Jan 2025 21:37:18 +0100
Labels:           app.kubernetes.io/name=cloudnative-pg
                  pod-template-hash=5d69854679
Annotations:      cni.projectcalico.org/containerID: 8f78b0d7236271c8a28874bb2e1d8f67cb1ae295342497a8cf1a33c747244ef5
                  cni.projectcalico.org/podIP: 10.42.0.29/32
                  cni.projectcalico.org/podIPs: 10.42.0.29/32
Status:           Pending
SeccompProfile:   RuntimeDefault
IP:               10.42.0.29
IPs:
  IP:           10.42.0.29
Controlled By:  ReplicaSet/cnpg-controller-manager-5d69854679
Containers:
  manager:
    Container ID:    
    Image:           cloudnative-pg:1.24.1
    Image ID:        
    Ports:           8080/TCP, 9443/TCP
    Host Ports:      0/TCP, 0/TCP
    SeccompProfile:  RuntimeDefault
    Command:
      /manager
    Args:
      controller
      --leader-elect
      --config-map-name=cnpg-controller-manager-config
      --secret-name=cnpg-controller-manager-config
      --webhook-port=9443
    State:          Waiting
      Reason:       ErrImagePull
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  200Mi
    Requests:
      cpu:      100m
      memory:   100Mi
    Liveness:   http-get https://:9443/readyz delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get https://:9443/readyz delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      OPERATOR_IMAGE_NAME:           cloudnative-pg:1.24.1
      OPERATOR_NAMESPACE:            cnpg-system (v1:metadata.namespace)
      MONITORING_QUERIES_CONFIGMAP:  cnpg-default-monitoring
    Mounts:
      /controller from scratch-data (rw)
      /run/secrets/cnpg.io/webhook from webhook-certificates (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-w5qgg (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  scratch-data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  webhook-certificates:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cnpg-webhook-cert
    Optional:    true
  kube-api-access-w5qgg:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  56s                default-scheduler  Successfully assigned cnpg-system/cnpg-controller-manager-5d69854679-h8zjl to vd013047
  Normal   Pulling    15s (x3 over 56s)  kubelet            Pulling image "cloudnative-pg:1.24.1"
  Warning  Failed     14s (x3 over 55s)  kubelet            Failed to pull image "cloudnative-pg:1.24.1": failed to pull and unpack image "docker.io/library/cloudnative-pg:1.24.1": failed to resolve reference "docker.io/library/cloudnative-pg:1.24.1": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
  Warning  Failed     14s (x3 over 55s)  kubelet            Error: ErrImagePull
  Normal   BackOff    2s (x3 over 55s)   kubelet            Back-off pulling image "cloudnative-pg:1.24.1"
  Warning  Failed     2s (x3 over 55s)   kubelet            Error: ImagePullBackOff

Observations:

1. The image cloudnative-pg:1.24.1 is successfully imported and tagged within the k8s.io namespace of the container runtime (confirmed via ctr -n k8s.io images list).
2. However, when the Kubernetes pod tries to pull the image, it attempts to fetch it from docker.io/library/cloudnative-pg:1.24.1, which fails due to the repository not existing or requiring authorization.

Question:

What am I missing in this process? Why is the image not being pulled from the local container runtime despite being imported and tagged correctly? Is there an additional configuration step required to make the image available to the rke2 cluster?

[brandond]

https://kubernetes.io/docs/concepts/containers/images/#image-pull-policy

Images are not "pulled from the local container runtime". The container runtime is the thing that pulls it if it is necessary, from a remote registry. There is nowhere other than the runtime for the image to go. If you're manually loading the image into the runtimes image store and want to skip pulling, then you need to say so.

Having it already on the node doesn't help if you don't set your image pull policy correctly in the pod spec.

[nskalis]

Thanks @brandond for your help! I'm encountering the following error

• after I have set imagePullPolicy: Never) when trying to use the OpenSSL container image (just another example)
• when describing the pod using kubectl:

Container image "openssl:latest" is not present with pull policy of Never

However, when I check the available images:

# ctr -n k8s.io images list  
REF                   TYPE                                       DIGEST                                                                  SIZE      PLATFORMS   LABELS  
openssl:latest        application/vnd.oci.image.manifest.v1+json sha256:d6a36e468ed88b9b362a794fb7b87386b2e5a5f057844fdedcb51bc564efbe39 8.5 MiB   linux/amd64 -  

And when I attempt to pull the image:

# ctr -n k8s.io images pull openssl:latest  
ctr: failed to resolve reference "openssl:latest": parse "dummy://openssl:latest": invalid port ":latest" after host  

Any ideas?

EDIT: on a side note, setting up a registry is not desired.

[brandond]

Using the latest tag is not recommended, as it has special semantics and default image pull behavior. This is covered in the upstream docs and in our docs for embedded registry mirror.

Also, you should use fully qualified image names. For example, instead of openssl:latest, use docker.io/library/openssl:latest. This avoids inconsistency due to use of an implicit registry (docker.io) and repository namespace (library).

[nskalis]

Thanks a lot! I'll definitely apply the second suggestion. Some images, like openssl, only provide the latest tag.

In my case, placing the .tar files in /var/lib/rancher/rke2/agent/images/ resolved the issue.

[brandond]

There is no openssl image on docker hub, which is probably part of what's confusing things: https://hub.docker.com/_/openssl

If you're going to use unpublished images, you should tag them as being from a fake/fictitious registry instead of docker.io. There is an example of this in our docs.

[nskalis]

I was initially looking into alpine/openssl, but I just discovered that they tag the latest version along with specific versions of the container image. I'll update my resources accordingly.

I read about the fictitious registry concept, but my focus is solely on locally available container images (that are copied from elsewhere). I've set up the following configuration, though I’m unsure if a DNS entry is required for registry.rke2.local.

/etc/rancher/rke2/registries.yaml

mirrors:
  registry.rke2.local:

/usr/local/lib/systemd/system/rke2-server.service

...
ExecStart=/usr/local/bin/rke2 server --write-kubeconfig-mode=644 --embedded-registry --system-default-registry "registry.rke2.local"
...

I’ll test this setup and share my findings in the discussion.