Why do registries in registries.yaml keep getting "v2" appened to end?

[davehouser1]

I have already read this page
Mainly this section

Default Endpoint Fallback
Containerd has an implicit "default endpoint" for all registries. The default endpoint is always tried as a last resort, even if there are other endpoints listed for that registry in registries.yaml. Rewrites are not applied to pulls against the default endpoint. For example, when pulling registry.example.com:5000/rancher/mirrored-pause:3.6, containerd will use a default endpoint of https://registry.example.com:5000/v2.
The default endpoint for docker.io is https://index.docker.io/v2.
The default endpoint for all other registries is https:///v2, where is the registry hostname and optional port.
In order to be recognized as a registry, the first component of the image name must contain at least one period or colon. For historical reasons, images without a registry specified in their name are implicitly identified as being from docker.io.

This makes sense for default path. But I am using a custom path.
Here is my /etc/rancher/k3s/registires.yaml file:

mirrors:
  our.internal.reg:
    endpoint:
      - https://our.internal.reg:443
configs:
  our.internal.reg:
    auth:
      email: "me@email.com"
      username: "me"
      token: "BigLongToken839483049830"
    tls:
      insecure_skip_verify: true

After I perform a systemctl restart k3s I see this in /var/lib/rancher/k3s/agent/etc/containerd/certs.d/artifact.it.att.com/hosts.toml

# File generated by k3s. DO NOT EDIT.

server = "https://our.internal.reg/v2"
capabilities = ["pull", "resolve", "push"]

skip_verify = true


[host]
[host."https://our.internal.reg:443/v2"]
  capabilities = ["pull", "resolve"]

But when trying to deploy my chart I see this error

  Failed to pull image "our.internal.reg/docker-hub/elastic/filebeat:8.17.0": 
  failed to pull and unpack image "our.internal.reg/docker-hub/elastic/filebeat:8.17.0": 
  failed to resolve reference "our.internal.reg/docker-hub/elastic/filebeat:8.17.0": 
  failed to authorize: failed to fetch anonymous token: 
  unexpected status from GET request to https://our.internal.reg/artifactory/api/docker/docker-hub/v2/token?scope=repository%3Adocker-hub%2Felastic%2Ffilebeat%3Apull&scope=repository%3Aelastic%2Ffilebeat%3Apull&service=our.internal.reg: 401

Is containerd trying to pull the image from the path https://our.internal.reg/artifactory/api/docker/docker-hub/v2? I dont manage our registry, but all the documents I read from it, v2 should not be in the path. Is this expected? How do I disable k3s from shoving in v2 in the URL path?

Note the registry is a Jfrog instance if that helps. Also note, I have tested access via docker pull using a config.json file and it works fine. Its not clear to me how I convert that config.json into a registry.yaml can anyone help?

[brandond]

The standard is for the root of the v2 registry API to be at /v2. If it is somewhere else, specify that path as part of your server URL. If you do not include the path, it is assumed that you simply forgot to include it - which is a common occurrence.

[brandond]

I will note that this error is NOT from your registry API:

failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://our.internal.reg/artifactory/api/docker/docker-hub/v2/token?scope=repository%3Adocker-hub%2Felastic%2Ffilebeat%3Apull&scope=repository%3Aelastic%2Ffilebeat%3Apull&service=our.internal.reg: 401

This error is from the auth service that your registry API referred containerd to in order to obtain an auth token. The 401 Unauthorized error indicates that you need to provide credentials.

[brandond]

Also note that there is no point in configuring this mirror entry:

mirrors:
  our.internal.reg:
    endpoint:
      - https://our.internal.reg:443

As noted in the docs you linked, https://our.internal.reg is the default endpoint for our.internal.reg. You should not include a mirrors entry for it. The presence of this entry with a port, while your configs section does NOT contain a port, is probably why the credentials are not being passed through. The authority from the mirrors entry (our.internal.reg:443) must match the key under configs if you want it to be used - but it does not. It is missing the port.

[davehouser1]

Thanks for the responses. So it seems like the authentication is the problem. Not the URL path.
I cleaned up the mirror section of repositories.yaml. But still it does not work.
Its not clear to me if containerd is using the correct creds. I tried using journalctl -fu containerd but no logs appear. k3s journalctl shows nothing helpful either.
In the mean time I am going to deploy a local registry see if I can get that working first. Will report back.

[brandond]

configs:
  our.internal.reg:
   auth:
     email: "me@email.com"
     username: "me"
     token: "BigLongToken839483049830"

Your auth section is also garbage. Are you just making up fields? Where did "email" and "token" come from? Have you read the docs?
https://docs.k3s.io/installation/private-registry#configs

[davehouser1]

hmmm one of those fields, I agree is not documented. email was from my ~/.docker/config.json. I was testing and left it in the config file that I posted in this thread. My mistake.
The other token I read from the documentation on that page you shared.
If you scroll up to the first example configs.auth.token is listed, I only realized now that scrolling down its not listed. Is that a typo?

Registries Configuration File
The file consists of two top-level keys, with subkeys for each registry:

mirrors:
  <REGISTRY>:
    endpoint:
      - https://<REGISTRY>/v2
configs:
  <REGISTRY>:
    auth:
      username: <BASIC AUTH USERNAME>
      password: <BASIC AUTH PASSWORD>
      token: <BEARER TOKEN>
    tls:
      ca_file: <PATH TO SERVER CA>
      cert_file: <PATH TO CLIENT CERT>
      key_file: <PATH TO CLIENT KEY>
      insecure_skip_verify: <SKIP TLS CERT VERIFICATION BOOLEAN>

I removed those fields. Now my config looks like this and it works now. Thanks for the help!

configs:
  our.internal.reg:
    auth:
      auth: "BigLongToken839483049830"
    tls:
      insecure_skip_verify: true

[brandond]

That is the Bearer Token auth value, as an alternative to Basic auth (username+password). You shouldn't use both. If your registry wants bearer token auth, that's the field to use.

Most things that take a token actually use this instead:

    auth:
      username: _json_token
      password: BIG-LONG-TOKEN-STRING

See https://github.com/containerd/containerd/blob/v2.0.2/docs/cri/registry.md#configure-registry-credentials-example---gcr-with-service-account-key-authentication

All of these things are just wired into fields in the containerd config, so the containerd docs are a good reference point if there is any question about how they work on the backend.