crypto/x509: FetchPEMRoots on macOS takes 20 seconds when there are a lot of trust settings

[tbodt]

Repro steps

First run this test program:

package main

import (
    "encoding/pem"
    "crypto/x509"
)

func main() {
	const certPEM = `
-----BEGIN CERTIFICATE-----
MIIDujCCAqKgAwIBAgIIE31FZVaPXTUwDQYJKoZIhvcNAQEFBQAwSTELMAkGA1UE
BhMCVVMxEzARBgNVBAoTCkdvb2dsZSBJbmMxJTAjBgNVBAMTHEdvb2dsZSBJbnRl
cm5ldCBBdXRob3JpdHkgRzIwHhcNMTQwMTI5MTMyNzQzWhcNMTQwNTI5MDAwMDAw
WjBpMQswCQYDVQQGEwJVUzETMBEGA1UECAwKQ2FsaWZvcm5pYTEWMBQGA1UEBwwN
TW91bnRhaW4gVmlldzETMBEGA1UECgwKR29vZ2xlIEluYzEYMBYGA1UEAwwPbWFp
bC5nb29nbGUuY29tMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEfRrObuSW5T7q
5CnSEqefEmtH4CCv6+5EckuriNr1CjfVvqzwfAhopXkLrq45EQm8vkmf7W96XJhC
7ZM0dYi1/qOCAU8wggFLMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAa
BgNVHREEEzARgg9tYWlsLmdvb2dsZS5jb20wCwYDVR0PBAQDAgeAMGgGCCsGAQUF
BwEBBFwwWjArBggrBgEFBQcwAoYfaHR0cDovL3BraS5nb29nbGUuY29tL0dJQUcy
LmNydDArBggrBgEFBQcwAYYfaHR0cDovL2NsaWVudHMxLmdvb2dsZS5jb20vb2Nz
cDAdBgNVHQ4EFgQUiJxtimAuTfwb+aUtBn5UYKreKvMwDAYDVR0TAQH/BAIwADAf
BgNVHSMEGDAWgBRK3QYWG7z2aLV29YG2u2IaulqBLzAXBgNVHSAEEDAOMAwGCisG
AQQB1nkCBQEwMAYDVR0fBCkwJzAloCOgIYYfaHR0cDovL3BraS5nb29nbGUuY29t
L0dJQUcyLmNybDANBgkqhkiG9w0BAQUFAAOCAQEAH6RYHxHdcGpMpFE3oxDoFnP+
gtuBCHan2yE2GRbJ2Cw8Lw0MmuKqHlf9RSeYfd3BXeKkj1qO6TVKwCh+0HdZk283
TZZyzmEOyclm3UGFYe82P/iDFt+CeQ3NpmBg+GoaVCuWAARJN/KfglbLyyYygcQq
0SgeDh8dRKUiaW3HQSoYvTvdTuqzwK4CXsr3b5/dAOY8uMuG/IAR3FgwTbZ1dtoW
RvOTa8hYiU6A475WuZKyEHcwnGYe57u2I2KbMgcKjPniocj4QzgYsVAVKW3IwaOh
yE+vPxsiUkvQHdO2fojCkY8jg70jxM+gu59tPDNbw3Uh/2Ij310FgTHsnGQMyA==
-----END CERTIFICATE-----`
    block, _ := pem.Decode([]byte(certPEM));
    if block == nil {
        panic("failed to parse cert")
    }
    cert, err := x509.ParseCertificate(block.Bytes)
    if err != nil {
        panic("failed to parse cert: " + err.Error())
    }
    if _, err := cert.Verify(x509.VerifyOptions{}); err != nil {
        panic("failed to verify cert: " + err.Error())
    }
}

Observe that it takes a fraction of a second.

Then set up a new macOS VM, download my trust settings from https://gist.githubusercontent.com/tbodt/57539144e5e1841d956706d0bece86a2/raw/37221140760d6af8556d8dcbdb66ab373cb4933f/trust.xml and import them using security trust-settings-import. Run the program again.

What did you expect to see?

The program should run in a fraction of a second.

What did you see instead?

The program takes 20 seconds.

When I run it on my computer and not in the testing VM, it takes 140 seconds, probably because I actually have the certificates that correspond to the trust settings. This causes it to always exceed the 30 second TLS connection timeout, which means any go program that uses https doesn't work.

What version of Go are you using (go version)?

go version go1.8 darwin/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/tbodt/go"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.8/libexec"
GOTOOLDIR="/usr/local/Cellar/go/1.8/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/5g/6q3kyyzs1xsg3kdy58zbg9940000gn/T/go-build077841286=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

[tbodt]

I would suggest using the system's certificate verification instead. This ends up moving the two minutes of work into the trustd daemon, which then creates a long-lived cache which every program can use.

Of course, the root issue is that this takes two minutes. I've had no luck getting Apple to talk to me about that.

[bradfitz]

To confirm, we're not talking about the cross-compiled CGO_ENABLED=0 code path, right?

[tbodt]

Yes, we're talking about the codepath with cgo. I'm not sure if the no-cgo codepath would be faster.

[adg]

I am encountering the 20 second delay only with cgo disabled. The first call to security trust-settings-export is what takes up the time. See upspin/upspin#413.

[adg]

In fact, it takes 20 seconds for security trust-settings-export to report (visible with GODEBUG=x509roots=1):

crypto/x509: exec ["/usr/bin/security" "trust-settings-export" "/var/folders/2r/2wscqnf957d_75ltvpj_n4zc0035qk/T/x509trustpolicy813089759/user"]: exit status 1, SecTrustSettingsCreateExternalRepresentation: No Trust Settings were found.

I wonder if anyone else has observed this? Seems like a long time to determine that there are no trust settings.

[tbodt]

To make it take 20 seconds with cgo enabled, you need to import thousands of trust settings. I've provided a file that you can use with security trust-settings-import. I don't recommend doing this outside a VM, as it's not easy to delete those trust settings.