This is a text-only version of the following page on https://raymii.org: --- Title : APT keeps complaining that the HTTPS certificate cannot be validated? Author : Remy van Elst Date : 11-01-2023 05:31 URL : https://raymii.org/s/blog/Syncthing_apt_repo_keeps_complaining_HTTPS_certificate_could_not_be_validated.html Format : Markdown/HTML --- ![gnutls logo][15] > The [GnuTLS][16] logo, the TLS library that apt uses. Recently a few of my Ubuntu 20.04 and Debian 11 servers failed to run an `apt update` because it insisted that the HTTPS certificate for a repository could not be validated, while `curl` on the same system had no issues connecting. Join me on a deep dive into certificate validation and troubleshooting `apt`, digging into the C++ source code for `apt` and `GnuTLS` and in the end, it turned out to be my own fault due to permission on a folder. However, the error messages were totally unhelpful resolving the mysterious validation problem. This article was written over the period of a few days, chronologically during troubleshooting.

Recently I removed all Google Ads from this site due to their invasive tracking, as well as Google Analytics. Please, if you found this content useful, consider a small donation using any of the options below:

I'm developing an open source monitoring app called Leaf Node Monitoring, for windows, linux & android. Go check it out!

Consider sponsoring me on Github. It means the world to me if you show your appreciation and you'll help pay the server costs.

You can also sponsor me by getting a Digital Ocean VPS. With this referral link you'll get $100 credit for 60 days.

Syncthing is a wonderful peace of software which syncs files to multiple servers / computers. I use it to keep a subset of my files synced among multiple computers and even on my servers sometimes to sync images. The version in the debian repositories is old and Syncthing provide an `apt` repository with clear copy and paste-able instructions, so I often use that and have it as an Ansible playbook so I can roll it out and add the peers automatically (the configuration is a simple XML file). The current process to add the syncthing apt repository, as [documented on their site][1] is the following, first get the GPG key, then add the apt repository marking it explicitly signed by the GPG key. Afterwards `update` and `install`: sudo curl -o /usr/share/keyrings/syncthing-archive-keyring.gpg https://syncthing.net/release-key.gpg echo "deb [signed-by=/usr/share/keyrings/syncthing-archive-keyring.gpg] https://apt.syncthing.net/ syncthing stable" | sudo tee /etc/apt/sources.list.d/syncthing.list sudo apt-get update sudo apt-get install syncthing However, the `apt update` step fails on my Ubuntu 20.04 and Debian 11 servers with the following error: Err:6 https://apt.syncthing.net syncthing Release Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown. Could not handshake: Error in the certificate verification. [IP: 143.244.196.6 443] The [guide][1] had a section on that specific error message: Server Certificate Verification Failed Especially for older distributions, your system TLS certificate store might be outdated. Since October 2021, a newer Let's Encrypt root certificate must be installed, or you may see an error similar to the following when running apt-get: E: Failed to fetch https://apt.syncthing.net/dists/syncthing/stable/binary-armhf/Packages server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none E: Some index files failed to download. They have been ignored, or old ones used instead. Please make sure you have the latest version of the ca-certificates package and try again: sudo apt-get update sudo apt-get install ca-certificates That suggestion was not helping, the `ca-certificates` package is up to date. Join me into a deep-dive on troubleshooting certificate validation paths for OpenSSL, curl, apt and GnuTLS on Debian and Ubuntu! ### Comparing the installed certificates As of writing this article, the certificate is signed by Comodo ZeroSSL, not Lets Encrypt: echo | openssl s_client -showcerts -connect apt.syncthing.net:443 2>&1 | grep -E "s:|i:" 0 s:CN = apt.syncthing.net i:C = AT, O = ZeroSSL, CN = ZeroSSL ECC Domain Secure Site CA 1 s:C = AT, O = ZeroSSL, CN = ZeroSSL ECC Domain Secure Site CA i:C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust ECC Certification Authority 2 s:C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust ECC Certification Authority i:C = GB, ST = Greater Manchester, L = Salford, O = Comodo CA Limited, CN = AAA Certificate Services The strange thing is that `curl` on the same systems has no issues with the certificate: curl -vI https://apt.syncthing.net Output: [...] * successfully set certificate verify locations: * CAfile: /etc/ssl/certs/ca-certificates.crt CApath: /etc/ssl/certs [...] * subject: CN=apt.syncthing.net * start date: Dec 1 00:00:00 2022 GMT * expire date: Mar 1 23:59:59 2023 GMT * subjectAltName: host "apt.syncthing.net" matched cert's "apt.syncthing.net" * issuer: C=AT; O=ZeroSSL; CN=ZeroSSL ECC Domain Secure Site CA * SSL certificate verify ok. Maybe `apt` uses a different root store? In [the apt changelog][6] I find some mentions on adding HTTPS support to the HTTP backend via GnuTLS and you used to have to install `apt-transport-https` but that is no longer needed since `apt` 1.5 as [the package states][7]: This is a dummy transitional package - https support has been moved into the apt package in 1.5. It can be safely removed. Looking through [the apt source code][8] I can see `GnuTLS` being used as the SSL backend. Installing `gnutls-bin` for the `certtool` command and using that to check the certificate using not OpenSSL but GnuTLS, shows it as valid: certtool --verify --infile chain.pem Output: Note that no verification profile was selected. In the future the medium profile will be enabled by default. Use --verify-profile low to apply the default verification of NORMAL priority string. Loaded system trust (125 CAs available) Subject: CN=apt.syncthing.net Issuer: CN=ZeroSSL ECC Domain Secure Site CA,O=ZeroSSL,C=AT Checked against: CN=ZeroSSL ECC Domain Secure Site CA,O=ZeroSSL,C=AT Signature algorithm: ECDSA-SHA384 Output: Verified. The certificate is trusted. Chain verification output: Verified. The certificate is trusted. Using `gnutls-cli apt.syncthing.net --print-cert` to get that cert also shows it being valid: Processed 125 CA certificate(s). Resolving 'apt.syncthing.net:443'... Connecting to '143.244.196.6:443'... - Certificate type: X.509 - Got a certificate list of 3 certificates. - Certificate[0] info: [...] - Status: The certificate is trusted. Just to be sure, let's also test with OpenSSL. Using [the openssl commands from here][2] I can also print the exact serial number and signatures/key id's of the certificates presented: OLDIFS=$IFS; IFS=':' certificates=$(openssl s_client -connect apt.syncthing.net:443 -showcerts -tlsextdebug 2>&1 Error("Certificate verification failed: %s", txt.data); } That [error code][10] says little: -348 GNUTLS_E_CERTIFICATE_VERIFICATION_ERROR Error in the certificate verification. But looking further in the apt source code, in `UnwrapTLS`, I can see another error message, `No system certificates available. Try installing ca-certificates`: // No CaInfo specified, use system trust store. err = gnutls_certificate_set_x509_system_trust(tlsFd->credentials); if (err == 0) Owner->Warning("No system certificates available. Try installing ca-certificates."); Maybe GnuTLS has trouble finding our system certificates. Lets figure out which file is used by default as a root trust store. The method `gnutls_certificate_set_x509_system_trust`, returns the number of certificates processed or a negative error code on error. Since we get exactly zero (0), it seems to be that GnuTLS is unable to load the system certificate store. I wonder where it tries to load them from. This method [seems to call][10] `gnutls_x509_trust_list_add_system_trust`, which in turn calls `add_system_trust`. The latter one is conditional `#ifdef`, but the implementation I'm looking at checks for `DEFAULT_TRUST_STORE_FILE`. That is not defined in code but is a `./configure` option, wonderful syntax but at least it has default filenames: dnl auto detect https://lists.gnu.org/archive/html/help-gnutls/2012-05/msg00004.html AC_ARG_WITH([default-trust-store-file], [AS_HELP_STRING([--with-default-trust-store-file=FILE], [use the given file default trust store])], with_default_trust_store_file="$withval", [if test "$build" = "$host" && test x$with_default_trust_store_pkcs11 = x && test x$with_default_trust_store_dir = x && test x$have_macosx = x;then for i in \ /etc/ssl/ca-bundle.pem \ /etc/ssl/certs/ca-certificates.crt \ /etc/pki/tls/cert.pem \ /usr/local/share/certs/ca-root-nss.crt \ /etc/ssl/cert.pem do if test -e "$i"; then with_default_trust_store_file="$i" break fi done fi] ) I'm not sure which compile option was used for the Debian packages, but I only have the file `/etc/ssl/certs/ca-certificates.crt` on my servers. Downloading the [source package][14] and looking at the `rules` file I can see that that is also the file used during compilation: CONFIGUREARGS = \ [...] --with-default-trust-store-file=/etc/ssl/certs/ca-certificates.crt \ [...] After all that searching through code we're still not any further. The files exist, another GnuTLS using-program can validate the certificate, so now what? ### My trusty old friend, strace As a last resort I tried `strace`. It shows you all the syscalls that are made. Lots of noise, but with a bit of filtering we can get useful information: strace -f -e stat,read,write,execve,openat -s 2048 apt update 2>&1 | less In that massive list of output I see this which seems relevant: [pid 2522130] execve("/usr/lib/apt/methods/https", ["/usr/lib/apt/methods/https"], 0x7fff668e0300 /* 23 vars */) = 0 This seems to be a URI handler for getting files from repositories. APT supports multiple schemes, methods as APT calls it, for repositories, such as http, ftp and cd-rom. It turns out that it did not have any options to tweak. Other strace output didn't have much to go on either. ### Disabling SSL, bad idea! Just disabling SSL certificate validation for this specific hostname is an option, since the packages are signed by a specific GPG key (mentioned in `sources.list`) and the `curl` command gave no certificate warning, and all manual troubleshooting shows me that, at this moment, the repository is signed by a trusted issuer. I don't like disabling verification, but since I was out of options, I added the [following options][5] to the specific `apt.conf.d` file: cat /etc/apt/apt.conf.d/80-ssl-exceptions File contents: Acquire::https::apt.syncthing.net::Verify-Peer "false"; Acquire::https::apt.syncthing.net::Verify-Host "false"; This results in no more errors during an `apt update`. But that was simply not an option for the long run. One more attempt, starting over a few days later. Sometimes it helps to take a step back and let it rest when you're knee-deep in a problem. ### Manually adding the entire chain for Apt Reading the manual page again for this config file I saw the `CaInfo` option. Earlier I tried that, by adding just the intermediate ZeroSSL CA file: # file: /etc/apt/apt.conf.d/80-ssl-exceptions Acquire::https::apt.syncthing.net::CaInfo "/etc/ssl/certs/apt.syncthing.net.chain.pem"; This failed with the following error: Could not load certificates from /etc/ssl/certs/apt.syncthing.net.chain.pem (CaInfo option): Error while reading file. [IP: 143.244.196.6 443] Trying the default system certificate store (`/etc/ssl/certs/ca-certificates.crt`) explicitly configured as `CaInfo` gave the same error. **Then it dawned on me**, another ansible playbook recently, a few weeks earlier, did stuff with the `/etc/ssl/` folder regarding certificate synchronization. Look at the permissions from a working server: remy@workingserver:~$ ls -lah /etc/ssl/ total 40K drwxr-xr-x 4 root root 4.0K Oct 26 2021 . drwxr-xr-x 94 root root 4.0K Jan 10 06:51 .. drwxr-xr-x 2 root root 16K Feb 15 2022 certs -rw-r--r-- 1 root root 11K Aug 24 2021 openssl.cnf drwx--x--- 2 root ssl-cert 4.0K Nov 26 2021 private Now compare that to all the non-working servers: root@non-working-server:~# ls -lah /etc/ssl/ total 48K drw-r--r-- 5 root root 4.0K Jan 8 13:00 . drwxr-xr-x 90 root root 4.0K Jan 8 12:25 .. drwxr-xr-x 2 root root 20K Jan 8 12:25 certs -rw-r--r-- 1 root root 11K May 30 2019 openssl.cnf drwx--x--- 2 root ssl-cert 4.0K Mar 19 2021 private Do you see it? # /etc/ssl/ drw-r--r-- 5 root root 4.0K Jan 8 13:00 . drwxr-xr-x 4 root root 4.0K Oct 26 2021 . Dropping to the user that `apt` uses shows me the error in more detail: su - _apt -s /bin/bash _apt@server:/$ ls -lah /etc/ssl/ ls: cannot access '/etc/ssl/certs': Permission denied ls: cannot access '/etc/ssl/.': Permission denied total 0 d????????? ? ? ? ? ? . d????????? ? ? ? ? ? certs _apt@server:/$ ls -lah /etc/ssl/certs ls: cannot access '/etc/ssl/certs': Permission denied The execute (`x`) bit for the folder `/etc/ssl/` was missing. Since you can't `execute` a directory, the execute bit has been put to better use. The execute bit on a directory allows you to access items that are inside the directory, even if you cannot list the directories contents. From the manpage: The letters rwxXst select file mode bits for the affected users: read (r), write (w), execute (or search for directories) (x), The folder on the broken servers has permission `644` and the working server has `755`. (The command `stat -c %a /path` shows you the permissions in octal form). The playbook I talked about earlier from a few weeks ago had given the `/etc/ssl` folder the wrong permissions. Lets try what happens with the correct permissions: chmod 755 /etc/ssl/ apt update Output: Hit:1 https://apt.syncthing.net syncthing InRelease Reading package lists... Done Building dependency tree Reading state information... Done All packages are up to date. No more errors! Re-doing the `strace` part and grepping for `Permission Denied` also showed the error: [pid 2526028] openat(AT_FDCWD, "/etc/ssl/certs/ca-certificates.crt", O_RDONLY) = -1 EACCES (Permission denied) I missed it due to there being so much other output. Knowing what to look for makes troubleshooting so much easier. ### Conclusion The error message `The certificate issuer is unknown` put me on the wrong track, going down a rabbit hole of SSL certificate validation and SSL library code, even into which SSL backend `apt` uses and how they validate certificates. Only after a few days and re-reading the man page, trying out a different option, I got a more clear error message: `Error while reading file`. After that it was an easy fix. Since I was running as `root` and was unaware `apt` drops privileges (to `_apt:nogroup`), I didn't look into the permissions right away. If the error message had been `permission denied while reading ca issuer file /etc/ssl/certs/ca-certificates.crt`, it would have been way more clear and easier to fix. But hey, I learned a bit more about how recent versions of `apt` handle SSL, took a look at the `apt` `c++` code and in the end banged my head against my desk since the issue was my fault all along. But that doesn't really matter since the journey towards the solution was valuable. [1]: https://web.archive.org/web/20221231011933/https://apt.syncthing.net/ [2]: /s/articles/OpenSSL_-_Get_all_certificates_from_a_website_in_plain_text.html [3]: https://web.archive.org/web/20230108204354/https://stackoverflow.com/questions/38899286/how-to-get-the-root-ca-certificate-fingerprint-using-openssl [4]: https://web.archive.org/web/20230108203241/https://manpages.ubuntu.com/manpages/focal/man5/apt.conf.5.html [5]: https://web.archive.org/web/20220516183420/https://manpages.ubuntu.com/manpages/focal/man1/apt-transport-https.1.html [6]: https://raw.githubusercontent.com/Debian/apt/main/debian/changelog [7]: https://web.archive.org/web/20230110062913/https://packages.debian.org/bullseye/apt-transport-https [8]: https://github.com/Debian/apt/blob/5919d2d18eac6e445a59da23246df94258e103eb/methods/connect.cc#L808 [9]: https://salsa.debian.org/apt-team/apt/-/blob/main/methods/connect.cc#L850 [10]: https://github.com/gnutls/gnutls/blob/4e5afffcba3e5f898eca7c450408605969135d48/lib/cert-cred-x509.c#L1281 [11]: https://gnutls.org/reference/gnutls-gnutls.html#gnutls-certificate-set-x509-system-trust [12]: https://github.com/gnutls/gnutls/blob/4e5afffcba3e5f898eca7c450408605969135d48/lib/system/certs.c#L372 [13]: https://github.com/gnutls/gnutls/blob/4e5afffcba3e5f898eca7c450408605969135d48/configure.ac#L1097 [14]: https://packages.ubuntu.com/source/focal/gnutls28 [15]: /s/inc/img/gnutls-logo-nobackground.png [16]: https://gnutls.org/gnutls-logo.html --- License: All the text on this website is free as in freedom unless stated otherwise. This means you can use it in any way you want, you can copy it, change it the way you like and republish it, as long as you release the (modified) content under the same license to give others the same freedoms you've got and place my name and a link to this site with the article as source. This site uses Google Analytics for statistics and Google Adwords for advertisements. You are tracked and Google knows everything about you. Use an adblocker like ublock-origin if you don't want it. All the code on this website is licensed under the GNU GPL v3 license unless already licensed under a license which does not allows this form of licensing or if another license is stated on that page / in that software: This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . Just to be clear, the information on this website is for meant for educational purposes and you use it at your own risk. I do not take responsibility if you screw something up. Use common sense, do not 'rm -rf /' as root for example. If you have any questions then do not hesitate to contact me. See https://raymii.org/s/static/About.html for details.