TLS 1.3 server problem with multiple private keys for the same certificate

[gilles-peskine-arm]

On a TLS server, you can configure multiple key-certificate combinations with mbedtls_ssl_conf_own_cert(). For each cipher suite that the client offers, among the ones that the server supports, the server walks through its list of key-certificate pairs and picks the first one that it thinks is compatible. As of Mbed TLS 3.6.4, TLS 1.2 and TLS 1.3 do this compatibility check differently:

• TLS 1.2 checks that both the private key and the public key in the certificate are compatible with the required algorithm. (It insists on randomized ECDSA when deterministic ECDSA would do but that's a separate problem.)
• TLS 1.3 only checks that the public key in the certificate is compatible with the required algorithm.

If a key-certificate pair looks compatible but turns out not to work, this causes the handshake to fail, even if a later key-certificate pair might work (possibly with a different cipher suite). This mainly means that passing a bad key-certificate pair will break handshakes even if another pair could succeed, but that's not a big issue: it's a misconfiguration, after all. But this can also break some valid configurations.

The TLS 1.3 behavior is a problem if you pass more than one key-certificate pair where the private key material and the certificate are the same, but the private key objects have different policies. For example, suppose that rsa15 and rsa21 are both PK objects wrapping PSA keys (MBEDTLS_PK_OPAQUE) with the same material, but the policy of rsa15 only allows PSA_ALG_RSA_PKCS1_V15_SIGN(PSA_ALG_ANY_HASH) while the policy of rsa21 only allows PSA_ALG_RSA_PSS(PSA_ALG_ANY_HASH). Then this works for both TLS 1.2 and TLS 1.3, because TLS 1.2 looks for a pair that can sign with PKCS#1v1.5, and TLS 1.3 picks the first RSA pair which happens to work:

mbedtls_ssl_conf_own_cert(ssl, rsa_cert, rsa21);
mbedtls_ssl_conf_own_cert(ssl, rsa_cert, rsa15);

But this breaks for TLS 1.3 if it chooses an RSA cipher suite, because it picks rsa_cert, then tries to sign with rsa15, but TLS 1.3 performs a PSS signature which fail due to the policy on rsa15.

mbedtls_ssl_conf_own_cert(ssl, rsa_cert, rsa15);
mbedtls_ssl_conf_own_cert(ssl, rsa_cert, rsa21);

In TLS 3.6, this isn't something most users would do, which explains why this has never been reported. This is more of a problem for Mbed TLS 4.x, where all keys are in PSA.

Reproducer: coming to #10217