[Bug]: Server does not utilize system CA Certificates through specified HTTP Proxy #46621

tunloop · 2024-07-18T20:26:43Z

⚠️ This issue respects the following points: ⚠️

This is a bug, not a question or a configuration/webserver/proxy issue.
This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
I agree to follow Nextcloud's Code of Conduct.

Bug description

Nextcloud outbound HTTP communications through forward HTTP proxy specified in config.php does not utilize the system ca store (on Debian: /etc/ssl/certs/ca-certificates.crt) even when CA file is explicitly specified in both FPM and CLI php.ini configuration files. (cURL error 35 - unknown CA)

Steps to reproduce

Specify HTTP proxy in config.php (ex. 'proxy' => '172.20.20.10:3142',)
Ensure php.ini in both FPM and CLI have CA crt file specified (ex. openssl.cafile=/etc/ssl/certs/ca-certificates.crt)
Test nextcloud's outbound connection by browsing to the app store in your web browser.
See error (in nextcloud logs and squid logs)

Expected behavior

Nextcloud, when using an HTTP proxy, should trust all CA certificates in system store.

Installation method

N/A

Nextcloud Server version

28

Operating system

Debian 12

PHP engine version

PHP 8.3

Web server

NGINX

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Unk/N/A

Are you using the Nextcloud Server Encryption module?

No

What user-backends are you using?

[ x] Default user-backend (database)
LDAP/ Active Directory
SSO - SAML
Other

Configuration report

{
    "system": {
        "proxy": "apt.example.com:3142",
        "proxyexclude": [
            "nextcloud.example.com"
        ],
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "172.20.20.5",
            "nextcloud.example.com",
            "172.20.20.6"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "29.0.3.4",
        "overwrite.cli.url": "http:\/\/nextcloud.example.com\/nextcloud",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "maintenance": false,
        "maintenance_window_start": 1,
        "theme": "",
        "loglevel": 2,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0,
            "dbindex": 0,
            "timeout": 1.5
        },
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "default_phone_region": "US",
        "allow_local_remote_servers": true,
        "mail_smtpmode": "sendmail",
        "mail_sendmailmode": "smtp",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "app_install_overwrite": [
            "breezedark"
        ],
        "app.mail.verify-tls-peer": false,
        "memories.exiftool_no_local": true,
        "memories.vod.path": "\/var\/www\/nextcloud\/apps\/memories\/bin-ext\/go-vod-amd64"
    }
}

List of activated Apps

Enabled:
  - admin_audit: 1.19.0
  - bruteforcesettings: 2.9.0
  - calendar: 4.7.12
  - checksum: 1.2.4
  - circles: 29.0.0-dev
  - cloud_federation_api: 1.12.0
  - collectives: 2.12.0
  - comments: 1.19.0
  - contactsinteraction: 1.10.0
  - dashboard: 7.9.0
  - dav: 1.30.1
  - external: 5.4.0
  - federatedfilesharing: 1.19.0
  - files: 2.1.0
  - files_downloadlimit: 2.0.0
  - files_pdfviewer: 2.10.0
  - files_reminders: 1.2.0
  - files_sharing: 1.21.0
  - files_trashbin: 1.19.0
  - files_versions: 1.22.0
  - logreader: 2.14.0
  - lookup_server_connector: 1.17.0
  - mail: 3.7.5
  - maps: 1.4.0
  - notes: 4.10.0
  - notifications: 2.17.0
  - oauth2: 1.17.0
  - password_policy: 1.19.0
  - photos: 2.5.0
  - privacy: 1.13.0
  - provisioning_api: 1.19.0
  - related_resources: 1.4.0
  - richdocuments: 8.4.3
  - riotchat: 0.17.6
  - serverinfo: 1.19.0
  - settings: 1.12.0
  - suspicious_login: 7.0.0
  - systemtags: 1.19.0
  - tables: 0.7.4
  - text: 3.10.1
  - theming: 2.4.0
  - theming_customcss: 1.16.0
  - twofactor_backupcodes: 1.18.0
  - updatenotification: 1.19.1
  - user_status: 1.9.0
  - viewer: 2.3.0
  - weather_status: 1.9.0
  - workflowengine: 2.11.0
Disabled:
  - activity: 2.21.1 (installed 2.13.4)
  - encryption: 2.17.0
  - federation: 1.19.0 (installed 1.10.1)
  - files_archive: 1.2.3 (installed 1.2.3)
  - files_external: 1.21.0
  - firstrunwizard: 2.18.0 (installed 2.9.0)
  - nextcloud_announcements: 1.18.0 (installed 1.9.0)
  - recommendations: 2.1.0 (installed 0.8.0)
  - sharebymail: 1.19.0 (installed 1.10.0)
  - support: 1.12.0 (installed 1.3.0)
  - survey_client: 1.17.0 (installed 1.8.0)
  - twofactor_totp: 11.0.0-dev
  - user_ldap: 1.20.0

Nextcloud Signing status

No errors have been found.

Nextcloud Logs - Limited (Github 65K character limit)

{"reqId":"LhQTAEFZsygTyEd2JOSG","level":2,"time":"2024-07-18T19:26:46+00:00","remoteAddr":"10.10.0.2","user":"bigdog","app":"appstoreFetcher","method":"GET","url":"/settings/apps","message":"Could not connect to appstore: cURL error 35: OpenSSL/3.0.13: error:8000000D:system library::Permission denied (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://apps.nextcloud.com/api/v1/apps.json","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0","version":"29.0.3.4","data":{"app":"appstoreFetcher"}}
{"reqId":"aptq5AybKgsT7H77SUWE","level":3,"time":"2024-07-18T19:32:28+00:00","remoteAddr":"10.10.0.2","user":"bigdog","app":"internet_connection_check","method":"GET","url":"/settings/ajax/checksetup","message":"Cannot connect to: www.nextcloud.com","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0","version":"29.0.3.4","exception":{"Exception":"GuzzleHttp\\Exception\\ConnectException","Message":"cURL error 35: OpenSSL/3.0.13: error:8000000D:system library::Permission denied (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://nextcloud.com/","Code":0,"Trace":[{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlFactory.php","line":158,"function":"createRejection","class":"GuzzleHttp\\Handler\\CurlFactory","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlFactory.php","line":110,"function":"finishError","class":"GuzzleHttp\\Handler\\CurlFactory","type":"::"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlHandler.php","line":47,"function":"finish","class":"GuzzleHttp\\Handler\\CurlFactory","type":"::"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Middleware.php","line":142,"function":"__invoke","class":"GuzzleHttp\\Handler\\CurlHandler","type":"->"},{"file":"/var/www/nextcloud/lib/private/Http/Client/DnsPinMiddleware.php","line":123,"function":"GuzzleHttp\\{closure}","class":"GuzzleHttp\\Middleware","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/PrepareBodyMiddleware.php","line":35,"function":"OC\\Http\\Client\\{closure}","class":"OC\\Http\\Client\\DnsPinMiddleware","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Middleware.php","line":31,"function":"__invoke","class":"GuzzleHttp\\PrepareBodyMiddleware","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/RedirectMiddleware.php","line":71,"function":"GuzzleHttp\\{closure}","class":"GuzzleHttp\\Middleware","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/RedirectMiddleware.php","line":107,"function":"__invoke","class":"GuzzleHttp\\RedirectMiddleware","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/RedirectMiddleware.php","line":73,"function":"checkRedirect","class":"GuzzleHttp\\RedirectMiddleware","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/promises/src/FulfilledPromise.php","line":41,"function":"GuzzleHttp\\{closure}","class":"GuzzleHttp\\RedirectMiddleware","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/promises/src/TaskQueue.php","line":48,"function":"GuzzleHttp\\Promise\\{closure}","class":"GuzzleHttp\\Promise\\FulfilledPromise","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php","line":248,"function":"run","class":"GuzzleHttp\\Promise\\TaskQueue","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php","line":224,"function":"invokeWaitFn","class":"GuzzleHttp\\Promise\\Promise","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php","line":269,"function":"waitIfPending","class":"GuzzleHttp\\Promise\\Promise","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php","line":226,"function":"invokeWaitList","class":"GuzzleHttp\\Promise\\Promise","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php","line":62,"function":"waitIfPending","class":"GuzzleHttp\\Promise\\Promise","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Client.php","line":189,"function":"wait","class":"GuzzleHttp\\Promise\\Promise","type":"->"},{"file":"/var/www/nextcloud/lib/private/Http/Client/Client.php","line":230,"function":"request","class":"GuzzleHttp\\Client","type":"->"},{"file":"/var/www/nextcloud/apps/settings/lib/SetupChecks/InternetConnectivity.php","line":85,"function":"get","class":"OC\\Http\\Client\\Client","type":"->"},{"file":"/var/www/nextcloud/apps/settings/lib/SetupChecks/InternetConnectivity.php","line":68,"function":"isSiteReachable","class":"OCA\\Settings\\SetupChecks\\InternetConnectivity","type":"->"},{"file":"/var/www/nextcloud/lib/private/SetupCheck/SetupCheckManager.php","line":51,"function":"run","class":"OCA\\Settings\\SetupChecks\\InternetConnectivity","type":"->"},{"file":"/var/www/nextcloud/apps/settings/lib/Controller/CheckSetupController.php","line":179,"function":"runAll","class":"OC\\SetupCheck\\SetupCheckManager","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php","line":232,"function":"check","class":"OCA\\Settings\\Controller\\CheckSetupController","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php","line":138,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/App.php","line":184,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/Route/Router.php","line":338,"function":"main","class":"OC\\AppFramework\\App","type":"::"},{"file":"/var/www/nextcloud/lib/base.php","line":1050,"function":"match","class":"OC\\Route\\Router","type":"->"},{"file":"/var/www/nextcloud/index.php","line":49,"function":"handleRequest","class":"OC","type":"::"}],"File":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlFactory.php","Line":210,"message":"Cannot connect to: www.nextcloud.com","exception":{},"CustomMessage":"Cannot connect to: www.nextcloud.com"}}

Additional info

No response

The text was updated successfully, but these errors were encountered:

tunloop · 2024-07-18T20:27:11Z

Additional information, since post was limited by characters

Jul 18 12:57:03 WebProxy squid[108450]: ERROR: failure while accepting a TLS connection on conn67 local=172.20.20.10:3142 remote=172.20.20.5:48320 FD 13 flags=1: 0x5a66a2522130*1
                                            current master transaction: master89
Jul 18 12:57:03 WebProxy squid[108450]: {"log":"squid-access","time":"18/Jul/2024:12:57:03 -0700","responseTime":21,"srcIp":"172.20.20.5","srcPort":"48320","destIp":"-","destPort":"-","userIdent":"-","user":"-","method":"CONNECT","httpVer":"1.1","url":"apps.nextcloud.com:443","referrer":"-","userAgent":"-","status":"200","reqAction":"NONE_NONE","reqStatus":"HIER_NONE","contentType":"-","bytes":103,"bytesIn":103,"bytesOut":0}

PCAP showing that nextcloud is closing the connection because it does not trust the CA cert.

Not an issue with Squid, curl on the nextcloud server using the system CA cert store functions just fine:



curl --cacert /etc/ssl/certs/ca-certificates.crt -vvv https://apps.nextcloud.com > /tmp/file
* Uses proxy env variable https_proxy == 'http://apt.example.com:3142'
Trying 172.20.20.10:3142...
* Connected to apt.example.com (172.20.20.10) port 3142 (#0)
* allocate connect buffer
* Establish HTTP proxy tunnel to apps.nextcloud.com:443
> CONNECT apps.nextcloud.com:443 HTTP/1.1
> Host: apps.nextcloud.com:443
> User-Agent: curl/7.88.1
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 Connection established
< 
* CONNECT phase completed
* CONNECT tunnel established, response 200
* ALPN: offers h2,http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [6 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2918 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [520 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: CN=apps.nextcloud.com
*  start date: Jul 13 23:41:26 2024 GMT
*  expire date: Jul 11 23:41:26 2034 GMT
*  subjectAltName: host "apps.nextcloud.com" matched cert's "apps.nextcloud.com"
*  issuer: C=US; ST=Washington; O=; CN=example.com; emailAddress=admin@example.com
*  SSL certificate verify ok.
* using HTTP/1.x
} [5 bytes data]
> GET / HTTP/1.1
> Host: apps.nextcloud.com
> User-Agent: curl/7.88.1
> Accept: */*
> 
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [233 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [233 bytes data]
* old SSL session ID is stale, removing
< HTTP/1.1 200 OK

joshtrichards · 2024-07-19T15:44:01Z

Related: nextcloud/documentation#11906

tunloop · 2024-07-19T20:59:04Z

Is the fix to add the certificate to the occ cert store?
sudo -u www-data php occ security:certificates:import /path/to/certificate

Is the related issue about just improving documentation, or making nextcloud utilize system certificate stores?

tunloop added 0. Needs triage Pending check for reproducibility or if it fits our roadmap bug labels Jul 18, 2024

szaimen added the 28-feedback label Jul 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: Server does not utilize system CA Certificates through specified HTTP Proxy #46621

[Bug]: Server does not utilize system CA Certificates through specified HTTP Proxy #46621

tunloop commented Jul 18, 2024 •

edited

Loading

tunloop commented Jul 18, 2024

joshtrichards commented Jul 19, 2024

tunloop commented Jul 19, 2024

[Bug]: Server does not utilize system CA Certificates through specified HTTP Proxy #46621

[Bug]: Server does not utilize system CA Certificates through specified HTTP Proxy #46621

Comments

tunloop commented Jul 18, 2024 • edited Loading

⚠️ This issue respects the following points: ⚠️

Bug description

Steps to reproduce

Expected behavior

Installation method

Nextcloud Server version

Operating system

PHP engine version

Web server

Database engine version

Is this bug present after an update or on a fresh install?

Are you using the Nextcloud Server Encryption module?

What user-backends are you using?

Configuration report

List of activated Apps

Nextcloud Signing status

Nextcloud Logs - Limited (Github 65K character limit)

Additional info

tunloop commented Jul 18, 2024

joshtrichards commented Jul 19, 2024

tunloop commented Jul 19, 2024

tunloop commented Jul 18, 2024 •

edited

Loading