The nginx registry cache cannot pull an image

[kubecto]

Hi, I use this docker registry UI and I have an issue...

Bug description

I have deployed a registry+ui mode and set the user login. I can access the http registry ui normally. At this time, it has user login authentication, but I want to configure a cache proxy to automatically log in users and pull the mirror. I hope that the cache proxy can be accessed through the domain name, but after I configure it through nginx, the header of the request forwarding cannot jump to the registry of the central control correctly

1, this is my central terminal registry+ui configuration

# cat registry.yml
services:
  registry-ui:
    image: joxit/docker-registry-ui:main
    restart: always
    ports:
      - 80:80
    environment:
      - SINGLE_REGISTRY=true
      - REGISTRY_TITLE=Docker Registry UI
      - DELETE_IMAGES=true
      - SHOW_CONTENT_DIGEST=true
      - NGINX_PROXY_PASS_URL=http://registry-server:5000
      - SHOW_CATALOG_NB_TAGS=true
      - CATALOG_MIN_BRANCHES=1
      - CATALOG_MAX_BRANCHES=1
      - TAGLIST_PAGE_SIZE=100
      - REGISTRY_SECURED=false
      - CATALOG_ELEMENTS_LIMIT=1000
    container_name: registry-ui

  registry-server:
    image: registry:2.8.2
    restart: always
    volumes:
      - /mnt/registry/data:/var/lib/registry
      - /mnt/registry/conf/config.yml:/etc/docker/registry/config.yml
      - /mnt/registry/conf/htpasswd:/etc/docker/registry/htpasswd
    container_name: registry-server

my registry config

# cat conf/config.yml
version: 0.1
log:
  fields:
    service: registry
storage:
  delete:
    enabled: true
  cache:
    blobdescriptor: inmemory
  filesystem:
    rootdirectory: /var/lib/registry
http:
  addr: :5000
  headers:
    X-Content-Type-Options: [nosniff]
    Access-Control-Allow-Origin: ['*']
    #Access-Control-Allow-Origin: ['https://mydown.test.cnm']
    Access-Control-Allow-Methods: ['HEAD', 'GET', 'OPTIONS', 'DELETE']
    Access-Control-Allow-Headers: ['Authorization', 'Accept', 'Cache-Control']
    Access-Control-Allow-Credentials: [true]
    Access-Control-Expose-Headers: ['Docker-Content-Digest']
auth:
  htpasswd:
    realm: registry
    path: /etc/docker/registry/htpasswd
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3

Startup script

cat start.sh
mkdir -p /mnt/registry/conf/

cp conf/config.yml /mnt/registry/conf/config.yml

docker run   --entrypoint htpasswd   httpd:2 -Bbn admin 9ijnmko0 >  /mnt/registry/conf/htpasswd

docker-compose -f registry.yml up -d

At this point I can normally access my ui page

2、From the proxy cache configuration, I also removed the user authentication, and added the proxy configuration to connect to the central control user name and password

 cat conf/config.yml
version: 0.1
log:
  fields:
    service: registry
storage:
  delete:
    enabled: true
  cache:
    blobdescriptor: inmemory
  filesystem:
    rootdirectory: /var/lib/registry
http:
  addr: :5000
  headers:
    X-Content-Type-Options: [nosniff]
    Access-Control-Allow-Origin: ['*']
    Access-Control-Allow-Methods: ['HEAD', 'GET', 'OPTIONS', 'DELETE']
    Access-Control-Allow-Headers: ['Authorization', 'Accept', 'Cache-Control']
    Access-Control-Allow-Credentials: [true]
    Access-Control-Expose-Headers: ['Docker-Content-Digest']
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3
proxy:
  remoteurl: http://10.102.98.20
  username: admin
  password: 9ijnmko0

 cat registry.yml
services:
  registry-ui:
    image: joxit/docker-registry-ui:main
    restart: always
    ports:
      - 80:80
    environment:
      - SINGLE_REGISTRY=true
      - REGISTRY_TITLE=Docker Registry UI
      - DELETE_IMAGES=true
      - SHOW_CONTENT_DIGEST=true
      - NGINX_PROXY_PASS_URL=http://registry-server:5000
      - NGINX_PROXY_HEADER_X_Forwarded_For=$$proxy_add_x_forwarded_for
      - SHOW_CATALOG_NB_TAGS=true
      - CATALOG_MIN_BRANCHES=1
      - CATALOG_MAX_BRANCHES=1
      - TAGLIST_PAGE_SIZE=100
      - REGISTRY_SECURED=false
      - CATALOG_ELEMENTS_LIMIT=1000
    container_name: registry-ui

  registry-server:
    image: registry:2.8.2
    restart: always
    volumes:
      - /mnt/registry/data:/var/lib/registry
      - /mnt/registry/conf/config.yml:/etc/docker/registry/config.yml
    container_name: registry-server

start script

cat start.sh
mkdir -p /mnt/registry/conf/

cp conf/config.yml /mnt/registry/conf/config.yml

docker-compose -f registry.yml up -d

3、from one of my nginx, this nginx I forward to the proxy cache node through the domain name

server {
        listen 443 ssl;
        server_name mydown.test.com;

        ssl_certificate "/usr/local/nginx/conf/ssl/fullchain.pem";
        ssl_certificate_key "/usr/local/nginx/conf/ssl/private.key";
        client_max_body_size 0;
        location / {
            proxy_pass http://10.102.98.22:80;
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }

        location /v2/ {
            proxy_pass http://10.102.98.22;
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;

            # These headers are required for Docker to trust the registry
            # certificate and enable client-side certificate validation
            proxy_ssl_verify on;
            proxy_ssl_trusted_certificate /usr/local/nginx/conf/ssl/fullchain.pem;
            proxy_ssl_session_reuse off;
        }
}

When I use the nginx domain name to fetch the image, the image cannot be found and I see the error log from registry, http.request.method=HEAD http.request.remoteaddr=10.102.98.22, And 10.102.98.20 was my central control address,

# docker pull registry.1stcs.cn/1stcs_containers/mysql:8.0.31
Error response from daemon: manifest for mydown.test.com/1stcs_containers/mysql:8.0.31 not found: manifest unknown: manifest unknown

Error message from registry

ERRO[0151] response completed with error                 err.code="manifest unknown" err.detail="unknown tag=8.0.31" err.message="manifest unknown" go.version=go1.20.8 http.request.host=mydown.test.com http.request.id=8c7b7d06-23cd-44e4-b303-3fefee8c5b25 http.request.method=GET http.request.remoteaddr=10.102.98.22 http.request.uri="/v2/1stcs_containers/mysql/manifests/8.0.31" http.request.useragent="docker/23.0.0 go/go1.19.5 git-commit/d7573ab kernel/3.10.0-1160.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/23.0.0 \(linux\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=6.38322ms http.response.status=404 http.response.written=96 vars.name="1stcs_containers/mysql" vars.reference=8.0.31

I also try to

https://github.com/distribution/distribution/issues/970#issuecomment-139414612 

and will proxy_set_header removed, these options are removed, also of no help

Expected behavior

I hope to use the proxy mydown.test.com to pull the mirror from the proxy cache and push the mirror through the central control

System information

-OS: [Centos 7.9]

• Browser:
• Name: [e.g. Chrome, Firefox]
• Docker registry UI:
• Version: [joxit/docker-registry-ui:main]
• Server: [docker]
• Docker version: [23.0.0]
• Docker registry ui tag: [joxit/docker-registry-ui:main]
• OS/Arch: [ linux/amd64]
• Tools: [e.g. docker-compose]

[Joxit]

Hi, your issue looks like a docker registry configuration itself, no link with my registry ui.

AFAIK the proxy option for docker registry is meant to be a fallback when the image is not found in your private one.

Please check their documentation: https://distribution.github.io/distribution/about/configuration/#proxy