TLS handshake cert error

[ianopolous]

I'm implementing a protocol using this and get this error in the handshake to an external server.

javax.net.ssl.SSLHandshakeException: QUICHE_ERR_TLS_FAIL: error:1000007e:SSL routines:OPENSSL_internal:CERT_CB_ERROR

The certificates are self signed, with an extension, and I have a custom trustmanager which is correctly verifying the certs and extension.

The same cert code works fine with TLS over TCP and netty. I've found this doc on the openssl error: https://www.openssl.org/docs/man1.0.2/man3/SSL_CTX_set_cert_cb.html which suggest some kind of error in the cert callback.

Any advice would be much appreciated, thank you for the awesome project.

[ianopolous]

Upon digging deeper, that link above says the callback should return 1 on success and 0 on error when verifying certs. But stepping through in the debugger I see BoringSSLCertificateVerifyCallback.java is returning BoringSSL.X509_V_OK, which is 0.

Am I barking up the wrong tree?

[ianopolous]

I've tried modifying the return value of that callback in BoringSSLCertificateVerifyCallback from 0 to 1 and then it gives a different error:
QUICHE_ERR_TLS_FAIL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
So seems like that's not the problem.

[normanmaurer]

can you share your impl ? Also are you sure its not throwing an exception etc ?

[ianopolous]

Sure, this is the relevant bit of code (Kotlin):
https://github.com/Peergos/jvm-libp2p/blob/feat/quic/src/main/kotlin/io/libp2p/transport/quic/QuicTransport.kt#L158
The TrustManager implementation is here (I checked it's returning fine without error):
https://github.com/Peergos/jvm-libp2p/blob/feat/quic/src/main/kotlin/io/libp2p/security/tls/TLSSecureChannel.kt#L193

[ianopolous]

There are examples of the kind of certs it's receiving here: It is always a single self signed cert with an extension:
https://github.com/Peergos/jvm-libp2p/blob/feat/quic/src/test/kotlin/io/libp2p/security/tls/CertificatesTest.kt

[ianopolous]

I've double checked that no exceptions are being thrown in any of my code. I also tried breaking on "any exception" and that didn't hit anything either, so it seems nothing in the JDK classes is throwing either.

[ianopolous]

I've pushed a minimalish test that reproduces it here:
https://github.com/Peergos/jvm-libp2p/blob/feat/quic/src/test/java/io/libp2p/core/QuicPingTestJava.java#L17
This test assumes the install-run-ipfs.sh in root has been run first to install and run the (Golang) server.

[ianopolous]

I've implemented the server side as well now, and got a pure java test that shows the same error, so no need for the external Golang server. I still might be doing something dumb in the client setup. Thank you for any suggestions.

This is the java client -> java server test:
https://github.com/Peergos/jvm-libp2p/blob/feat/quic/src/test/java/io/libp2p/core/QuicServerTestJava.java#L15

[ianopolous]

I've tracked this down to us using Ed25519 certificates, so it was Boringssl that was throwing the exception because of an unsupported key type (BoringSSLCertificateCallback line 161). It seems like they don't support Ed25519 certificates:
cloudflare/boring#113
cloudflare/quiche#1482

[normanmaurer]

So I guess can close this one ?

[ianopolous]

The 2nd linked issue suggests it's possible to add ed25519 to the boringssl config in the signature algorithms list. It would be great to enable it by default if possible.

[normanmaurer]

@ianopolous I wonder if its a good idea to enable it by default. I guess we could allow to enable it tho .

[ianopolous]

It's more modern and secure than all the other algorithms. So I don't see a reason not to enable it.

[normanmaurer]

@ianopolous I am mostly "caution" here as BoringSSL doesn't do it by default. Usually these people have a good reason for doing so :)

[ianopolous]

Sure, it's a good question. Just having the option at all would be great.