TLS 1.2 can use the wrong algorithm with RSA keys

[gilles-peskine-arm]

Summary

An RSA key passed to mbedtls_ssl_conf_own_cert() for TLS 1.2 must have PKCS#1v1.5 as its primary algorithm. This is the case by default. But if the key is set up for PSS/OAEP as the primary algorithm, the key can't be used for TLS 1.2. This affects both MBEDTLS_PK_RSA keys whose underlying context has the padding mode set to MBEDTLS_RSA_PKCS_V21, and MBEDTLS_PK_OPAQUE RSA keys where the required PKCS#1v1.5 algorithm has been set with psa_set_key_enrollment_algorithm() rather than psa_set_key_algorithm().

Arguably the ”enrollment“ algorithm is not intended for such usage. However, the semantics of the function suggests that the two algorithms in the policy are interchangeable. This is not true through the PK interface.

This bug does not affect TLS 1.3. In TLS 1.2, it only affects RSA keys, not ECC keys (used for ECDH or ECDSA).

Detailed analysis

The TLS 1.2 code uses mbedtls_pk_sign to create signatures and mbedtls_pk_decrypt to decrypt ciphertexts. Under the hood, this function decides which algorithm to use based on the private key object. This can be the wrong algorithm, depending on how the private key was set up.

• For a transparent RSA key (backed by mbedtls_rsa_context), the algorithm is based on the padding mode selected when setting up the context. This is PKCS#1v1.5 signature and PKCS#1v1.5 decryption by default, which is fine for TLS 1.2. But if mbedtls_rsa_set_padding(rsa, MBEDTLS_RSA_PKCS_V21, ...) has been called on the RSA context, TLS 1.2 performs an OAEP decryption or a PSS signature instead.

  This is not a security violation in the crypto layer since the padding mode is only a default and can be bypassed.

• For an opaque RSA key (wrapping a PSA key), the algorithm is read from the primary algorithm in the PSA key policy. However, the PSA key may be acceptable only because the necessary algorithm has been set as the enrollment algorithm in the policy. In this case, TLS 1.2 performs the primary algorithm, and either fails (if this is the wrong category of algorithms) or obtains the wrong result (if this is the wrong algorithm in the right category).

  There is no security violation in the crypto layer since the key is capable of both the required algorithm and the actual algorithm.

• For an ECC key, the signature is always ECDSA, so there is no problem.

Using the wrong algorithm is fortunately not a security issue since only the following three bad scenarios are possible:

• TLS 1.2 wants to perform an RSA PKCS#1v1.5 signature, but ends up sending a PSS signature to the peer instead. The PSS signature contains a unique random challenge, so this is technically an existential forgery oracle, but there's no way for an adversary to obtain the PSS signature of a chosen message. Furthermore, being able to obtain a PSS signature is likely to be harmless.
• TLS 1.2 wants to perform an RSA PKCS#1v1.5 decryption, but ends up performing an OAEP decryption instead. This will fail unless the peer sends a valid OAEP ciphertext. If the peer sends a valid OAEP ciphertext that decrypts to a valid plaintext, the TLS 1.2 will use the resulting plaintext.
• TLS 1.2 wants to perform an RSA operation, but ends up trying to perform an operation with an algorithm in the wrong category. This is a runtime failure which is caught properly.

All of these problems happen systematically when the problematic key is used. Therefore any problem would be caught immediately when trying to use the key. This bug doesn't create dangerous latent situations.

I've only analyzed and tested the server side. I believe the situation is the same for client signatures, but I haven't checked.

Workaround

If you want a RSA key to be usable for TLS 1.2 (and optionally TLS 1.3), do not set the padding mode to MBEDTLS_RSA_PKCS_V21. The default padding mode MBEDTLS_RSA_PKCS_V15 works fine. In particular, a key loaded with mbedtls_pk_parse_key() or mbedtls_pk_parse_key_file() works fine.

If you want an opaque RSA key to be usable for both TLS 1.2 and TLS 1.3, use

psa_set_key_algorithm(key_attributes, PSA_ALG_RSA_PKCS1V15_SIGN(PSA_ALG_ANY_HASH));
psa_set_key_enrollment_algorithm(key_attributes, PSA_ALG_RSA_PSS(PSA_ALG_ANY_HASH));

and not the other way round.

It is not possible to use a single opaque RSA key for both decryption-based cipher suites (RSA-WITH-... without ECDH(E) or DH(E)) and signature-based RSA cipher suites ((EC)DH(E)-RSA-WITH-...) in TLS 1.2. Use separate key objects instead. Note that decryption-based cipher suites are deprecated.

Loosely related, but not the same root cause: #10220

Reproducer

Reproducer for the opaque key cases:
https://github.com/gilles-peskine-arm/mbedtls/tree/tls12-pk-psa-alg2-3.6

I haven't tried to reproduce the transparent case in 3.6 (too much hassle to set up), but I have a reproducer in my work in progress for #10160.

Urgency

Given that this issue only affects unusual setups, and hasn't been reported by a user, it's likely that we'll never fix it. I only noticed it because I was adding tests for mbedtls_ssl_conf_own_cert() and one of the test cases failed.