SSL problem with OTP 27.1.3

[egze]

Describe the bug
On OTP 27.1.3 connection to an API endpoint stopped working, and it worked fine with OTP 26.

Not really sure where to start looking.

iex(22)> :ssl.connect('THE_SERVICE', 443, ssl_opts)
warning: single-quoted strings represent charlists. Use ~c"" if you indeed want a charlist or use "" instead
└─ iex:22:14

10:12:43.936 mfa=:ssl_handshake.path_validation_alert/3 [notice] TLS :client: In state :certify at ssl_handshake.erl:2199 generated CLIENT ALERT: Fatal - Unsupported Certificate
 - {:key_usage_mismatch,
 {{:Extension, {2, 5, 29, 15}, true, [:keyCertSign, :cRLSign]},
  {:Extension, {2, 5, 29, 37}, false,
   [{1, 3, 6, 1, 5, 5, 7, 3, 2}, {1, 3, 6, 1, 5, 5, 7, 3, 1}]}}}
{:error,
 {:tls_alert,
  {:unsupported_certificate,
   ~c"TLS client: In state certify at ssl_handshake.erl:2199 generated CLIENT ALERT: Fatal - Unsupported Certificate\n {key_usage_mismatch,{{'Extension',{2,5,29,15},true,[keyCertSign,cRLSign]},\n                      {'Extension',{2,5,29,37},\n                                   false,\n                                   [{1,3,6,1,5,5,7,3,2},\n                                    {1,3,6,1,5,5,7,3,1}]}}}"}}}

Affected versions
27.1.3

[IngelaAndin]

This behavior is related to fixing a vulnerability of misusage of certificate purposes. That was previous too relaxed.

From RFC:

https://datatracker.ietf.org/doc/html/rfc5280#section-4.2.1.12

If a certificate contains both a key usage extension and an extended
key usage extension, then both extensions MUST be processed
independently and the certificate MUST only be used for a purpose
consistent with both extensions.

These extended key-usages {1, 3, 6, 1, 5, 5, 7, 3, 2}, {1, 3, 6, 1, 5, 5, 7, 3, 1} are the oids for id-kp-serverAuth and id-kp-clientAuth
and the are consistent with key use: digitalSignature and keyAgreement but your key uses are: keyCertSign, :cRLSig

Also from RFC:

id-kp-serverAuth             OBJECT IDENTIFIER ::= { id-kp 1 }
   -- TLS WWW server authentication
   -- Key usage bits that may be consistent: digitalSignature,
   -- keyEncipherment or keyAgreement

   id-kp-clientAuth             OBJECT IDENTIFIER ::= { id-kp 2 }
   -- TLS WWW client authentication
   -- Key usage bits that may be consistent: digitalSignature
   -- and/or keyAgreement

If the certificate failing is a CA cert it can also be allowed if the CA cert also assert the anyExtendedKeyUsage and the ssl configuration adds the option allow_any_ca_purpose is configured to be true.

So the proper thing to so is to get you certificates correct, and your last option would be to have your own verify_fun that
makes an exception if you consider it not be a problem, and then going against the RFC turns out to be a problem it is your responsibility.

[tarzan]

I am lost here: why does Erlang :ssl not allow a connection with the server here while I do not get any error messages when I use any other tool or browser?

Isn't Erlang :ssl too strict here and aren't we suffering because everyone else can connect without a problem with a server that has such a certificate while we using the latest Erlang OTP can't?

[tarzan]

@IngelaAndin how can I sidestep this in a minimal way? Do I have to feed :hackney a specific argument or function? Do you maybe have an example? I am struggling with this because I am mainly an Elixir guy.

[IngelaAndin]

@tarzan I can not answer your question unless you tell me how your certificate that you think should work but does not looks.

[tarzan]

I am trying to connect to server controlled by the Dutch government, in order to fetch FederationMetadata to be used for single sign on.

I get an error message similar to the OP:

TLS client: In state certify at ssl_handshake.erl:2199 generated CLIENT ALERT: Fatal - Unsupported Certificate
{key_usage_mismatch,{{'Extension',{2,5,29,15},true,[keyCertSign,cRLSign]},
                                        {'Extension',{2,5,29,37},
                                        false,
                                        [{1,3,6,1,5,5,7,3,2},
                                        {1,3,6,1,5,5,7,3,1}]}}

This is the server and certificate:

❯ openssl s_client -connect wsfed.sson.sscict.rijksdienst.nl:443 -showcerts

Connecting to 147.181.98.166
CONNECTED(00000005)
depth=2 C=US, O=Entrust, Inc., OU=See www.entrust.net/legal-terms, OU=(c) 2009 Entrust, Inc. - for authorized use only, CN=Entrust Root Certification Authority - G2
verify return:1
depth=1 C=US, O=Entrust, Inc., OU=See www.entrust.net/legal-terms, OU=(c) 2014 Entrust, Inc. - for authorized use only, CN=Entrust Certification Authority - L1M
verify return:1
depth=0 C=NL, ST=Zuid-Holland, L=Den Haag, jurisdictionC=NL, O=SSC-ICT, businessCategory=Government Entity, serialNumber=KvK 59115289, CN=wsfed.ssc-ict.overheid-i.nl
verify return:1
---
Certificate chain
 0 s:C=NL, ST=Zuid-Holland, L=Den Haag, jurisdictionC=NL, O=SSC-ICT, businessCategory=Government Entity, serialNumber=KvK 59115289, CN=wsfed.ssc-ict.overheid-i.nl
   i:C=US, O=Entrust, Inc., OU=See www.entrust.net/legal-terms, OU=(c) 2014 Entrust, Inc. - for authorized use only, CN=Entrust Certification Authority - L1M
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
   v:NotBefore: May 30 08:38:20 2024 GMT; NotAfter: May 30 08:38:19 2025 GMT
-----BEGIN CERTIFICATE-----
MIIIjDCCB3SgAwIBAgIQPbPAwpR5bRT9x2PQONTd3TANBgkqhkiG9w0BAQsFADCB
ujELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUVudHJ1c3QsIEluYy4xKDAmBgNVBAsT
H1NlZSB3d3cuZW50cnVzdC5uZXQvbGVnYWwtdGVybXMxOTA3BgNVBAsTMChjKSAy
MDE0IEVudHJ1c3QsIEluYy4gLSBmb3IgYXV0aG9yaXplZCB1c2Ugb25seTEuMCwG
A1UEAxMlRW50cnVzdCBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eSAtIEwxTTAeFw0y
NDA1MzAwODM4MjBaFw0yNTA1MzAwODM4MTlaMIG3MQswCQYDVQQGEwJOTDEVMBMG
A1UECBMMWnVpZC1Ib2xsYW5kMREwDwYDVQQHEwhEZW4gSGFhZzETMBEGCysGAQQB
gjc8AgEDEwJOTDEQMA4GA1UEChMHU1NDLUlDVDEaMBgGA1UEDxMRR292ZXJubWVu
dCBFbnRpdHkxFTATBgNVBAUTDEt2SyA1OTExNTI4OTEkMCIGA1UEAxMbd3NmZWQu
c3NjLWljdC5vdmVyaGVpZC1pLm5sMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIIC
CgKCAgEAq1gGR4ibD2+oQOCV5DeR1+VJu1FUWXJDfgZDvUo8lcSgYwEjb6yEk2Ow
EIvAFWxDNH1nfZV+Gne+K3RcNJlYacaT+bY306IPy9HnyonIYt9iGHbU2uL8iw/P
N4u7AU/SfWb4bfBlhLnS09qdf3N/EYxiVvI3K+MEoxbWq0f7PbojECN4ZlBc2Im2
FakmXeIvQLhVZatT3GxmPpUx/1snosLsYatAvFmynMqdC/mNEgu9lNkgGbhj9MTR
09V8PiHcgVmPhlI6FK8j/tDAotYBYDpdePA136aQKAxwvi1u9g7+RQ1axNyp5snq
JCFoqh0esNpdl2t7pnzqtt3kTZYzChSS68+hgEhW5C9e38U5xMG8cGKD+xzi3xoa
LX1N/TXStbm64dP50OKxfCw69OkxDpmfLKRwa0x3yX9sjvvqdS9XbosMz2Xn3/Jn
oedmg7W+M8J050wElDjxnANRa84gcAzwjJs1zLrvMl5ihyNGPuCsaUpeuNdug5uO
1XxY50LcEZojgfMpwYvHeqI3oSVUzIldIJdgztaCwy4Npf+pBuKUp7hJCiO6XtK7
5p6s14xU1usA6Z9QExl2NfZsj6ENJn7v1yAvoGYtDlun0rOPK3mWLuylPvB2+nI8
dfCcgjx7iOiA/a5N9AxzpGwFQZwAnZcQ/oX+4Z7rJb3S+F/sZ1sCAwEAAaOCA40w
ggOJMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFENcTSg0Ay4AEejhxFMqdXA/Q4Ow
MB8GA1UdIwQYMBaAFMP30LUqMK2vDZEhcDlU3byJcMc6MGgGCCsGAQUFBwEBBFww
WjAjBggrBgEFBQcwAYYXaHR0cDovL29jc3AuZW50cnVzdC5uZXQwMwYIKwYBBQUH
MAKGJ2h0dHA6Ly9haWEuZW50cnVzdC5uZXQvbDFtLWNoYWluMjU2LmNlcjAzBgNV
HR8ELDAqMCigJqAkhiJodHRwOi8vY3JsLmVudHJ1c3QubmV0L2xldmVsMW0uY3Js
MIGbBgNVHREEgZMwgZCCG3dzZmVkLnNzYy1pY3Qub3ZlcmhlaWQtaS5ubIIkY2Vy
dGF1dGgud3NmZWQuc3NjLWljdC5vdmVyaGVpZC1pLm5sgiB3c2ZlZC5zc29uLnNz
Y2ljdC5yaWprc2RpZW5zdC5ubIIpY2VydGF1dGgud3NmZWQuc3Nvbi5zc2NpY3Qu
cmlqa3NkaWVuc3QubmwwDgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUF
BwMBBggrBgEFBQcDAjBLBgNVHSAERDBCMAcGBWeBDAEBMDcGCmCGSAGG+mwKAQIw
KTAnBggrBgEFBQcCARYbaHR0cHM6Ly93d3cuZW50cnVzdC5uZXQvcnBhMIIBfgYK
KwYBBAHWeQIEAgSCAW4EggFqAWgAdgDm0jFjQHeMwRBBBtdxuc7B0kD2loSG+7qH
Mh39HjeOUAAAAY/IpmwdAAAEAwBHMEUCIGe+GIn2pG2+0m/v/ekxYQ4ShkKzzQxL
SIxr4/bNF334AiEA8uGSadS4gjXVOEpaOcPTX2kOmE69/r1LRBq6vyQXgNwAdgCi
4wrkRe+9rZt+OO1HZ3dT14JbhJTXK14bLMS5UKRH5wAAAY/IpmwzAAAEAwBHMEUC
IQD3wO44CaSGVkr1N37oUaqm+7DilsubwqN0ypQ/LLvxdQIgXEM8fjwLTCx1fSjv
rJLsQz/u0IY5yaMmPA1P/llBMZQAdgBOdaMnXJoQwzhbbNTfP1LrHfDgjhuNacCx
+mSxYpo53wAAAY/IpmxJAAAEAwBHMEUCIFiQ5UnT4rWWaaUUd6TWeO/kBydV8blG
jwi0AXpj01pQAiEAnGDZRV/xwUNrJ17JGNqyAm3NqdGdyg3q+2F2GHGgf1MwDQYJ
KoZIhvcNAQELBQADggEBAEZBiGk+3qsbyAhPnlP5O4nb9jq3JI0fyE3W7O1IFU6e
jNilmal5V4BQbUvppwRs3WXdt0rDZ4IB8W6Fy+T6PcSzPBu6Y4EEQi/aDC0iNnfe
6Sy/DvOfQ0SW7zc0rOoZxfJsWsM1t9wicOLBdHm9RtdT6Vh1HZKWUyE5rILryNlF
wK9kiPNGSdfkc8lrp+0AgrqyV5miyquW733ljMglSiVeaZM+qlKE+7jUm1H9uzfX
8Q7VssQsP2SZD6gUEJtXRzB7odtqDNQG/b6ENvzDn35LEmNNFjhj1OEXpN/F5r1U
SGXefxi1KQbDYzOL/6XibJT3CNC8vScgrfuI1EcQuAI=
-----END CERTIFICATE-----
 1 s:C=US, O=Entrust, Inc., OU=See www.entrust.net/legal-terms, OU=(c) 2014 Entrust, Inc. - for authorized use only, CN=Entrust Certification Authority - L1M
   i:C=US, O=Entrust, Inc., OU=See www.entrust.net/legal-terms, OU=(c) 2009 Entrust, Inc. - for authorized use only, CN=Entrust Root Certification Authority - G2
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Dec 15 15:25:03 2014 GMT; NotAfter: Oct 15 15:55:03 2030 GMT
-----BEGIN CERTIFICATE-----
MIIFLTCCBBWgAwIBAgIMYaHn0gAAAABR02amMA0GCSqGSIb3DQEBCwUAMIG+MQsw
CQYDVQQGEwJVUzEWMBQGA1UEChMNRW50cnVzdCwgSW5jLjEoMCYGA1UECxMfU2Vl
IHd3dy5lbnRydXN0Lm5ldC9sZWdhbC10ZXJtczE5MDcGA1UECxMwKGMpIDIwMDkg
RW50cnVzdCwgSW5jLiAtIGZvciBhdXRob3JpemVkIHVzZSBvbmx5MTIwMAYDVQQD
EylFbnRydXN0IFJvb3QgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkgLSBHMjAeFw0x
NDEyMTUxNTI1MDNaFw0zMDEwMTUxNTU1MDNaMIG6MQswCQYDVQQGEwJVUzEWMBQG
A1UEChMNRW50cnVzdCwgSW5jLjEoMCYGA1UECxMfU2VlIHd3dy5lbnRydXN0Lm5l
dC9sZWdhbC10ZXJtczE5MDcGA1UECxMwKGMpIDIwMTQgRW50cnVzdCwgSW5jLiAt
IGZvciBhdXRob3JpemVkIHVzZSBvbmx5MS4wLAYDVQQDEyVFbnRydXN0IENlcnRp
ZmljYXRpb24gQXV0aG9yaXR5IC0gTDFNMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A
MIIBCgKCAQEA0IHBOSPCsdHs91fdVSQ2kSAiSPf8ylIKsKs/M7WwhAf23056sPuY
Ij0BrFb7cW2y7rmgD1J3q5iTvjOK64dex6qwymmPQwhqPyK/MzlG1ZTy4kwFItln
gJHxBEoOm3yiydJs/TwJhL39axSagR3nioPvYRZ1R5gTOw2QFpi/iuInMlOZmcP7
lhw192LtjL1JcdJDQ6Gh4yEqI3CodT2ybEYGYW8YZ+QpfrI8wcVfCR5uRE7sIZlY
FUj0VUgqtzS0BeN8SYwAWN46lsw53GEzVc4qLj/RmWLoquY0djGqr3kplnjLgRSv
adr7BLlZg0SqCU+01CwBnZuUMWstoc/B5QIDAQABo4IBKzCCAScwDgYDVR0PAQH/
BAQDAgEGMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDATASBgNVHRMBAf8E
CDAGAQH/AgEAMDMGCCsGAQUFBwEBBCcwJTAjBggrBgEFBQcwAYYXaHR0cDovL29j
c3AuZW50cnVzdC5uZXQwMAYDVR0fBCkwJzAloCOgIYYfaHR0cDovL2NybC5lbnRy
dXN0Lm5ldC9nMmNhLmNybDA7BgNVHSAENDAyMDAGBFUdIAAwKDAmBggrBgEFBQcC
ARYaaHR0cDovL3d3dy5lbnRydXN0Lm5ldC9ycGEwHQYDVR0OBBYEFMP30LUqMK2v
DZEhcDlU3byJcMc6MB8GA1UdIwQYMBaAFGpyJnrQHu995ztpUdRsjZ+QEmarMA0G
CSqGSIb3DQEBCwUAA4IBAQC0h8eEIhopwKR47PVPG7SEl2937tTPWa+oQ5YvHVje
pvMVWy7ZQ5xMQrkXFxGttLFBx2YMIoYFp7Qi+8VoaIqIMthx1hGOjlJ+Qgld2dnA
DizvRGsf2yS89byxqsGK5Wbb0CTz34mmi/5e0FC6m3UAyQhKS3Q/WFOv9rihbISY
Jnz8/DVRZZgeO2x28JkPxLkJ1YXYJKd/KsLak0tkuHB8VCnTglTVz6WUwzOeTTRn
4Dh2ZgCN0C/GqwmqcvrOLzWJ/MDtBgO334wlV/H77yiI2YIowAQPlIFpI+CRKMVe
1QzX1CA778n4wI+nQc1XRG5sZ2L+hN/nYNjvv9QiHg3n
-----END CERTIFICATE-----
---
Server certificate
subject=C=NL, ST=Zuid-Holland, L=Den Haag, jurisdictionC=NL, O=SSC-ICT, businessCategory=Government Entity, serialNumber=KvK 59115289, CN=wsfed.ssc-ict.overheid-i.nl
issuer=C=US, O=Entrust, Inc., OU=See www.entrust.net/legal-terms, OU=(c) 2014 Entrust, Inc. - for authorized use only, CN=Entrust Certification Authority - L1M
---
No client certificate CA names sent
Peer signing digest: SHA512
Peer signature type: RSA
Server Temp Key: ECDH, secp384r1, 384 bits
---
SSL handshake has read 4321 bytes and written 501 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Protocol: TLSv1.2
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 3A507CAB0E14710F23198A26745D1896B6E6D609361F3CBC678B75A446ED68A7
    Session-ID-ctx:
    Master-Key: DAD68D1240732A49BE3CD204E22ECA0D3D875CE07BD89D2792B9F75622ADA7913C57C1D2B86B6B0670D9100D6E70B03C
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1736512645
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes
---

[hkrutzer]

I've been following this issue, as we've had to rollback an Erlang release because we need to connect to a Dutch govt site as well (https://chat.ind.nl). I've not looked into it deeply yet, and it may well be that there is a certificate issue, but I second the observation that there seems to be no other tool that rejects this certificate, not do any of the TLS quality scanners.

[IngelaAndin]

Well, this seems really strange. I was using OpenSSL extensively to try and double check the understanding of how this should work. But right now I do not seem to be able to reproduce that. So I will have to investigate this further next week.

[egze]

We ended up implementing our own verify_fun. But to me, the new logic is also too strict. We are connecting to a bank using their certificate, it is not a problem for other customers with other programming languages. And it is highly unlikely that the bank will change the certificate.

[IngelaAndin]

I just realized that a difference between this cert and some other certs that we saw failing is that this cert has keyCertSign bit set.
And although RFC 5280 is quite unclear about this I feel that this might logical be warrant to not issue this warning, especially as OpenSSL and others seem to accept it. But will look into it more on Monday.

[tarzan]

Thanks a lot for all the replies, glad you're willing to look into this some more :)

@egze if you're willing, I would be very much interested in seeing the custom verify_fun you wrote

[maysam]

@egze 👍

[dmorneau]

And although RFC 5280 is quite unclear about this I feel that this might logical be warrant to not issue this warning, especially as OpenSSL and others seem to accept it. But will look into it more on Monday.

Hi and thanks for looking into it! My team supports that accepting it is correct for these reasons:

• The language in RFC 5280 regarding EKU was based on an assumption that “In general, this extension will only appear in end entity certificates”.
• In practice the EKU on an intermediate CA certificate is used to constrain the type of certificates issued by that intermediate, not the usage of the intermediate certificate itself. See for example the Mozilla Root Store Policy under "5.3.1 Technically Constrained", and this discussion in mozilla.dev.security.policy.
• The CA/Browser Forum’s Baseline Requirements allow intermediate certificates to have id-kp-serverAuth EKU without a compatible KU. They:
  • Specifically and explicitly require keyUsage and extendedKeyUsage (see 7.1.2.6.1)
  • Specifically require crlSign on the intermediate KU (7.1.2.10.7)
  • Specifically require id-kp-serverAuth in the EKU (7.1.2.10.6)
• While many intermediate certificates in the WebPKI ecosystem happen to include the digitalSignature KU, which is compatible with the id-kp-serverAuth EKU and therefore makes them valid in the OTP 27.1.3 implementation, this is only a coincidence. The digitalSignature is intended to express that the key may be used to sign OCSP request, not that it may be used for server authentication in a TLS connection. See Baseline Requirements (7.1.2.10.7):

If a CA Certificate does not assert the digitalSignature bit, the CA Private Key MUST NOT be used to sign an OCSP Response.

This means the concrete result of the current (OTP 27.1.3) behavior is to reject intermediate certificates whose key isn't used to sign OSCP responses, which is clearly not the intent of RFC 5280.

(Edit: this is also a good reference explaining OpenSSL's behavior: https://docs.openssl.org/master/man1/openssl-verification-options/#extended-key-usage)

[IngelaAndin]

• The CA/Browser Forum’s Baseline Requirements allow intermediate certificates to have id-kp-serverAuth EKU without a compatible KU. They:

  • Specifically and explicitly require keyUsage and extendedKeyUsage (see 7.1.2.6.1)
  • Specifically require crlSign on the intermediate KU (7.1.2.10.7)
  • Specifically require id-kp-serverAuth in the EKU (7.1.2.10.6)

Well yes this part support the current suggested solution in #8806

(Edit: this is also a good reference explaining OpenSSL's behavior: https://docs.openssl.org/master/man1/openssl-verification-options/#extended-key-usage)

Thanks, I already read this several times.

The alternative would be to make the exception in the id-kp-serverAuth and id-kp-clientAuth in the ssl (TLS) applications own verify_fun, but I am leaning to that this check belongs in public_key and that it will not affect other than TLS applications.
No waiting for input form my team.