fleet-server fips140=only test failures

[michel-laterman]

When testing on a Linux VM with microsoft/go and a FIPS provider, running CGO_ENABLED=1 FIPS=true make test-unit will result in the following tests failing.
These are all tests around our handling of certs, so this is unlikely to be an issue with the binary and probably is an issue with our test code:

=== RUN   Test_server_ClientCert
=== RUN   Test_server_ClientCert/no_client_certs
    server.go:98: {"level":"info","message":"Listening on localhost:41397"}
    server.go:151: {"level":"error","message":"http: panic serving 127.0.0.1:34782: EVP_KDF_derive\nopenssl error(s):\nerror:1C800069:Provider routines::invalid key length\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\n\t../providers/implementations/kdfs/hkdf.c:163\ngoroutine 644 [running]:\nnet/http.(*conn).serve.func1()\n\t/usr/local/go/src/net/http/server.go:1947 +0x10a\npanic({0x206b4a0?, 0xc000049b00?})\n\t/usr/local/go/src/runtime/panic.go:787 +0x132\ncrypto/tls/internal/tls13.ExpandLabel[...](0xc000307140, {0xc001904100, 0x20, 0x20}, {0x228ae16, 0x2}, {0x0, 0x0, 0x0}, 0xc)\n\t/usr/local/go/src/crypto/tls/internal/tls13/tls13.go:41 +0x413\ncrypto/tls.(*cipherSuiteTLS13).trafficKey(0x320e7a0, {0xc001904100, 0x20, 0x20})\n\t/usr/local/go/src/crypto/tls/key_schedule.go:29 +0x1bd\ncrypto/tls.(*halfConn).setTrafficSecret(0xc00024f688, 0x320e7a0, 0x2, {0xc001904100, 0x20, 0x20})\n\t/usr/local/go/src/crypto/tls/conn.go:234 +0x106\ncrypto/tls.(*serverHandshakeStateTLS13).sendServerParameters(0xc000307610)\n\t/usr/local/go/src/crypto/tls/handshake_server_tls13.go:779 +0xa7e\ncrypto/tls.(*serverHandshakeStateTLS13).handshake(0xc000307610)\n\t/usr/local/go/src/crypto/tls/handshake_server_tls13.go:80 +0xc5\ncrypto/tls.(*Conn).serverHandshake(0xc00024f508, {0x2511018, 0xc00018ed20})\n\t/usr/local/go/src/crypto/tls/handshake_server.go:56 +0x25d\ncrypto/tls.(*Conn).handshakeContext(0xc00024f508, {0x2510fe0, 0xc001f65740})\n\t/usr/local/go/src/crypto/tls/conn.go:1568 +0x603\ncrypto/tls.(*Conn).HandshakeContext(...)\n\t/usr/local/go/src/crypto/tls/conn.go:1508\nnet/http.(*conn).serve(0xc001847dd0, {0x2510fe0, 0xc00205fd10})\n\t/usr/local/go/src/net/http/server.go:1971 +0x433\ncreated by net/http.(*Server).Serve in goroutine 634\n\t/usr/local/go/src/net/http/server.go:3454 +0x8ca\n"}
    server_test.go:169:
        	Error Trace:	/home/ubuntu/fleet-server/internal/pkg/api/server_test.go:169
        	Error:      	Received unexpected error:
        	            	Get "https://localhost:41397/api/status": EOF
        	Test:       	Test_server_ClientCert/no_client_certs
=== RUN   Test_server_ClientCert/valid_client_certs
    server.go:98: {"level":"info","message":"Listening on localhost:43115"}
    server.go:151: {"level":"error","message":"http: panic serving 127.0.0.1:37202: EVP_KDF_derive\nopenssl error(s):\nerror:0308010C:digital envelope routines::unsupported\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\n\t../crypto/evp/evp_fetch.c:349\nerror:1C800069:Provider routines::invalid key length\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\n\t../providers/implementations/kdfs/hkdf.c:163\ngoroutine 638 [running]:\nnet/http.(*conn).serve.func1()\n\t/usr/local/go/src/net/http/server.go:1947 +0x10a\npanic({0x206b4a0?, 0xc0004191f0?})\n\t/usr/local/go/src/runtime/panic.go:787 +0x132\ncrypto/tls/internal/tls13.ExpandLabel[...](0xc000025140, {0xc0003e6980, 0x20, 0x20}, {0x228ae16, 0x2}, {0x0, 0x0, 0x0}, 0xc)\n\t/usr/local/go/src/crypto/tls/internal/tls13/tls13.go:41 +0x413\ncrypto/tls.(*cipherSuiteTLS13).trafficKey(0x320e7a0, {0xc0003e6980, 0x20, 0x20})\n\t/usr/local/go/src/crypto/tls/key_schedule.go:29 +0x1bd\ncrypto/tls.(*halfConn).setTrafficSecret(0xc0000cc508, 0x320e7a0, 0x2, {0xc0003e6980, 0x20, 0x20})\n\t/usr/local/go/src/crypto/tls/conn.go:234 +0x106\ncrypto/tls.(*serverHandshakeStateTLS13).sendServerParameters(0xc000025610)\n\t/usr/local/go/src/crypto/tls/handshake_server_tls13.go:779 +0xa7e\ncrypto/tls.(*serverHandshakeStateTLS13).handshake(0xc000025610)\n\t/usr/local/go/src/crypto/tls/handshake_server_tls13.go:80 +0xc5\ncrypto/tls.(*Conn).serverHandshake(0xc0000cc388, {0x2511018, 0xc001755810})\n\t/usr/local/go/src/crypto/tls/handshake_server.go:56 +0x25d\ncrypto/tls.(*Conn).handshakeContext(0xc0000cc388, {0x2510fe0, 0xc002128690})\n\t/usr/local/go/src/crypto/tls/conn.go:1568 +0x603\ncrypto/tls.(*Conn).HandshakeContext(...)\n\t/usr/local/go/src/crypto/tls/conn.go:1508\nnet/http.(*conn).serve(0xc0001cd5f0, {0x2510fe0, 0xc0020140f0})\n\t/usr/local/go/src/net/http/server.go:1971 +0x433\ncreated by net/http.(*Server).Serve in goroutine 650\n\t/usr/local/go/src/net/http/server.go:3454 +0x8ca\n"}
    server_test.go:241:
        	Error Trace:	/home/ubuntu/fleet-server/internal/pkg/api/server_test.go:241
        	Error:      	Received unexpected error:
        	            	Get "https://localhost:43115/api/status": EOF
        	Test:       	Test_server_ClientCert/valid_client_certs
=== RUN   Test_server_ClientCert/invalid_client_certs
    server.go:98: {"level":"info","message":"Listening on localhost:42115"}
    server.go:151: {"level":"error","message":"http: panic serving 127.0.0.1:58804: EVP_KDF_derive\nopenssl error(s):\nerror:1C800069:Provider routines::invalid key length\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\n\t../providers/implementations/kdfs/hkdf.c:163\ngoroutine 678 [running]:\nnet/http.(*conn).serve.func1()\n\t/usr/local/go/src/net/http/server.go:1947 +0x10a\npanic({0x206b4a0?, 0xc0002ca010?})\n\t/usr/local/go/src/runtime/panic.go:787 +0x132\ncrypto/tls/internal/tls13.ExpandLabel[...](0xc000307140, {0xc0003e76c0, 0x20, 0x20}, {0x228ae16, 0x2}, {0x0, 0x0, 0x0}, 0xc)\n\t/usr/local/go/src/crypto/tls/internal/tls13/tls13.go:41 +0x413\ncrypto/tls.(*cipherSuiteTLS13).trafficKey(0x320e7a0, {0xc0003e76c0, 0x20, 0x20})\n\t/usr/local/go/src/crypto/tls/key_schedule.go:29 +0x1bd\ncrypto/tls.(*halfConn).setTrafficSecret(0xc000148508, 0x320e7a0, 0x2, {0xc0003e76c0, 0x20, 0x20})\n\t/usr/local/go/src/crypto/tls/conn.go:234 +0x106\ncrypto/tls.(*serverHandshakeStateTLS13).sendServerParameters(0xc000307610)\n\t/usr/local/go/src/crypto/tls/handshake_server_tls13.go:779 +0xa7e\ncrypto/tls.(*serverHandshakeStateTLS13).handshake(0xc000307610)\n\t/usr/local/go/src/crypto/tls/handshake_server_tls13.go:80 +0xc5\ncrypto/tls.(*Conn).serverHandshake(0xc000148388, {0x2511018, 0xc000134550})\n\t/usr/local/go/src/crypto/tls/handshake_server.go:56 +0x25d\ncrypto/tls.(*Conn).handshakeContext(0xc000148388, {0x2510fe0, 0xc000d471a0})\n\t/usr/local/go/src/crypto/tls/conn.go:1568 +0x603\ncrypto/tls.(*Conn).HandshakeContext(...)\n\t/usr/local/go/src/crypto/tls/conn.go:1508\nnet/http.(*conn).serve(0xc000155950, {0x2510fe0, 0xc00042cc60})\n\t/usr/local/go/src/net/http/server.go:1971 +0x433\ncreated by net/http.(*Server).Serve in goroutine 657\n\t/usr/local/go/src/net/http/server.go:3454 +0x8ca\n"}
    server.go:74: {"level":"warn","error":"close tcp 127.0.0.1:42115: use of closed network connection","message":"server.Run: error while closing listener."}
=== RUN   Test_server_ClientCert/valid_client_certs_no_certs_requested
    server.go:98: {"level":"info","message":"Listening on localhost:39959"}
    server.go:151: {"level":"error","message":"http: panic serving 127.0.0.1:42778: EVP_KDF_derive\nopenssl error(s):\nerror:1C800069:Provider routines::invalid key length\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\n\t../providers/implementations/kdfs/hkdf.c:163\ngoroutine 666 [running]:\nnet/http.(*conn).serve.func1()\n\t/usr/local/go/src/net/http/server.go:1947 +0x10a\npanic({0x206b4a0?, 0xc000261940?})\n\t/usr/local/go/src/runtime/panic.go:787 +0x132\ncrypto/tls/internal/tls13.ExpandLabel[...](0xc000307140, {0xc001904ee0, 0x20, 0x20}, {0x228ae16, 0x2}, {0x0, 0x0, 0x0}, 0xc)\n\t/usr/local/go/src/crypto/tls/internal/tls13/tls13.go:41 +0x413\ncrypto/tls.(*cipherSuiteTLS13).trafficKey(0x320e7a0, {0xc001904ee0, 0x20, 0x20})\n\t/usr/local/go/src/crypto/tls/key_schedule.go:29 +0x1bd\ncrypto/tls.(*halfConn).setTrafficSecret(0xc0000ccc08, 0x320e7a0, 0x2, {0xc001904ee0, 0x20, 0x20})\n\t/usr/local/go/src/crypto/tls/conn.go:234 +0x106\ncrypto/tls.(*serverHandshakeStateTLS13).sendServerParameters(0xc000307610)\n\t/usr/local/go/src/crypto/tls/handshake_server_tls13.go:779 +0xa7e\ncrypto/tls.(*serverHandshakeStateTLS13).handshake(0xc000307610)\n\t/usr/local/go/src/crypto/tls/handshake_server_tls13.go:80 +0xc5\ncrypto/tls.(*Conn).serverHandshake(0xc0000cca88, {0x2511018, 0xc0003ee6e0})\n\t/usr/local/go/src/crypto/tls/handshake_server.go:56 +0x25d\ncrypto/tls.(*Conn).handshakeContext(0xc0000cca88, {0x2510fe0, 0xc000df1380})\n\t/usr/local/go/src/crypto/tls/conn.go:1568 +0x603\ncrypto/tls.(*Conn).HandshakeContext(...)\n\t/usr/local/go/src/crypto/tls/conn.go:1508\nnet/http.(*conn).serve(0xc0010fb440, {0x2510fe0, 0xc000d47a10})\n\t/usr/local/go/src/net/http/server.go:1971 +0x433\ncreated by net/http.(*Server).Serve in goroutine 662\n\t/usr/local/go/src/net/http/server.go:3454 +0x8ca\n"}
    server_test.go:396:
        	Error Trace:	/home/ubuntu/fleet-server/internal/pkg/api/server_test.go:396
        	Error:      	Received unexpected error:
        	            	Get "https://localhost:39959/api/status": EOF
        	Test:       	Test_server_ClientCert/valid_client_certs_no_certs_requested
--- FAIL: Test_server_ClientCert (4.89s)
    --- FAIL: Test_server_ClientCert/no_client_certs (0.63s)
    wserver.go:74: {"level":"warn","error":"close tcp 127.0.0.1:41397: use of closed network connection","message":"server.Run: error while closing listener."}
    --- FAIL: Test_server_ClientCert/valid_client_certs (0.76s)
    wserver.go:74: {"level":"warn","error":"close tcp 127.0.0.1:43115: use of closed network connection","message":"server.Run: error while closing listener."}
    --- PASS: Test_server_ClientCert/invalid_client_certs (1.03s)
    --- FAIL: Test_server_ClientCert/valid_client_certs_no_certs_requested (1.28s)

=== RUN   TestAPMHTTPTransportOptions/custom_cert
    instrumentation_test.go:94: start test server to verify TLSClientConfig...
2025/03/21 19:20:33 http: panic serving 127.0.0.1:43508: EVP_KDF_derive
openssl error(s):
error:1C800069:Provider routines::invalid key length
	../providers/implementations/kdfs/hkdf.c:163
goroutine 177 [running]:
net/http.(*conn).serve.func1()
	/usr/local/go/src/net/http/server.go:1947 +0x10a
panic({0xfb4720?, 0xc0003e6100?})
	/usr/local/go/src/runtime/panic.go:787 +0x132
crypto/tls/internal/tls13.ExpandLabel[...](0xc0000ed140, {0xc0003be1e0, 0x20, 0x20}, {0x10a2d39, 0x2}, {0x0, 0x0, 0x0}, 0xc)
	/usr/local/go/src/crypto/tls/internal/tls13/tls13.go:41 +0x413
crypto/tls.(*cipherSuiteTLS13).trafficKey(0x170b040, {0xc0003be1e0, 0x20, 0x20})
	/usr/local/go/src/crypto/tls/key_schedule.go:29 +0x1bd
crypto/tls.(*halfConn).setTrafficSecret(0xc0000aa508, 0x170b040, 0x2, {0xc0003be1e0, 0x20, 0x20})
	/usr/local/go/src/crypto/tls/conn.go:234 +0x106
crypto/tls.(*serverHandshakeStateTLS13).sendServerParameters(0xc0000ed610)
	/usr/local/go/src/crypto/tls/handshake_server_tls13.go:779 +0xa7e
crypto/tls.(*serverHandshakeStateTLS13).handshake(0xc0000ed610)
	/usr/local/go/src/crypto/tls/handshake_server_tls13.go:80 +0xc5
crypto/tls.(*Conn).serverHandshake(0xc0000aa388, {0x11d7d88, 0xc000194050})
	/usr/local/go/src/crypto/tls/handshake_server.go:56 +0x25d
crypto/tls.(*Conn).handshakeContext(0xc0000aa388, {0x11d7d50, 0xc0003e42a0})
	/usr/local/go/src/crypto/tls/conn.go:1568 +0x603
crypto/tls.(*Conn).HandshakeContext(...)
	/usr/local/go/src/crypto/tls/conn.go:1508
net/http.(*conn).serve(0xc000188240, {0x11d7d50, 0xc0003e4120})
	/usr/local/go/src/net/http/server.go:1971 +0x433
created by net/http.(*Server).Serve in goroutine 178
	/usr/local/go/src/net/http/server.go:3454 +0x8ca
    instrumentation_test.go:112:
        	Error Trace:	/home/ubuntu/fleet-server/internal/pkg/config/instrumentation_test.go:112
        	Error:      	Received unexpected error:
        	            	Get "https://127.0.0.1:43423": EOF
        	Test:       	TestAPMHTTPTransportOptions/custom_cert

=== RUN   TestClientCerts/no_certs
2025/03/21 19:20:39 http: panic serving 127.0.0.1:43928: EVP_KDF_derive
openssl error(s):
error:0308010C:digital envelope routines::unsupported
	../crypto/evp/evp_fetch.c:349
error:1C800069:Provider routines::invalid key length
	../providers/implementations/kdfs/hkdf.c:163
goroutine 97 [running]:
net/http.(*conn).serve.func1()
	/usr/local/go/src/net/http/server.go:1947 +0x10a
panic({0x18c4820?, 0xc000037850?})
	/usr/local/go/src/runtime/panic.go:787 +0x132
crypto/tls/internal/tls13.ExpandLabel[...](0xc00003b140, {0xc00002b380, 0x20, 0x20}, {0x1a29d2f, 0x2}, {0x0, 0x0, 0x0}, 0xc)
	/usr/local/go/src/crypto/tls/internal/tls13/tls13.go:41 +0x413
crypto/tls.(*cipherSuiteTLS13).trafficKey(0x25aacc0, {0xc00002b380, 0x20, 0x20})
	/usr/local/go/src/crypto/tls/key_schedule.go:29 +0x1bd
crypto/tls.(*halfConn).setTrafficSecret(0xc0001a3a08, 0x25aacc0, 0x2, {0xc00002b380, 0x20, 0x20})
	/usr/local/go/src/crypto/tls/conn.go:234 +0x106
crypto/tls.(*serverHandshakeStateTLS13).sendServerParameters(0xc00003b610)
	/usr/local/go/src/crypto/tls/handshake_server_tls13.go:779 +0xa7e
crypto/tls.(*serverHandshakeStateTLS13).handshake(0xc00003b610)
	/usr/local/go/src/crypto/tls/handshake_server_tls13.go:80 +0xc5
crypto/tls.(*Conn).serverHandshake(0xc0001a3888, {0x1bed290, 0xc000362230})
	/usr/local/go/src/crypto/tls/handshake_server.go:56 +0x25d
crypto/tls.(*Conn).handshakeContext(0xc0001a3888, {0x1bed258, 0xc0003721e0})
	/usr/local/go/src/crypto/tls/conn.go:1568 +0x603
crypto/tls.(*Conn).HandshakeContext(...)
	/usr/local/go/src/crypto/tls/conn.go:1508
net/http.(*conn).serve(0xc0001b4870, {0x1bed258, 0xc0003720f0})
	/usr/local/go/src/net/http/server.go:1971 +0x433
created by net/http.(*Server).Serve in goroutine 8
	/usr/local/go/src/net/http/server.go:3454 +0x8ca
    client_test.go:63:
        	Error Trace:	/home/ubuntu/fleet-server/internal/pkg/es/client_test.go:63
        	Error:      	Received unexpected error:
        	            	EOF
        	Test:       	TestClientCerts/no_certs
=== RUN   TestClientCerts/uses_certs
2025/03/21 19:20:39 http: panic serving 127.0.0.1:38270: EVP_KDF_derive
openssl error(s):
error:1C800069:Provider routines::invalid key length
	../providers/implementations/kdfs/hkdf.c:163
goroutine 13 [running]:
net/http.(*conn).serve.func1()
	/usr/local/go/src/net/http/server.go:1947 +0x10a
panic({0x18c4820?, 0xc000037b10?})
	/usr/local/go/src/runtime/panic.go:787 +0x132
crypto/tls/internal/tls13.ExpandLabel[...](0xc0003df140, {0xc00002b580, 0x20, 0x20}, {0x1a29d2f, 0x2}, {0x0, 0x0, 0x0}, 0xc)
	/usr/local/go/src/crypto/tls/internal/tls13/tls13.go:41 +0x413
crypto/tls.(*cipherSuiteTLS13).trafficKey(0x25aacc0, {0xc00002b580, 0x20, 0x20})
	/usr/local/go/src/crypto/tls/key_schedule.go:29 +0x1bd
crypto/tls.(*halfConn).setTrafficSecret(0xc0000af688, 0x25aacc0, 0x2, {0xc00002b580, 0x20, 0x20})
	/usr/local/go/src/crypto/tls/conn.go:234 +0x106
crypto/tls.(*serverHandshakeStateTLS13).sendServerParameters(0xc0003df610)
	/usr/local/go/src/crypto/tls/handshake_server_tls13.go:779 +0xa7e
crypto/tls.(*serverHandshakeStateTLS13).handshake(0xc0003df610)
	/usr/local/go/src/crypto/tls/handshake_server_tls13.go:80 +0xc5
crypto/tls.(*Conn).serverHandshake(0xc0000af508, {0x1bed290, 0xc0000fa870})
	/usr/local/go/src/crypto/tls/handshake_server.go:56 +0x25d
crypto/tls.(*Conn).handshakeContext(0xc0000af508, {0x1bed258, 0xc00048ec60})
	/usr/local/go/src/crypto/tls/conn.go:1568 +0x603
crypto/tls.(*Conn).HandshakeContext(...)
	/usr/local/go/src/crypto/tls/conn.go:1508
net/http.(*conn).serve(0xc0000ee3f0, {0x1bed258, 0xc00048eb70})
	/usr/local/go/src/net/http/server.go:1971 +0x433
created by net/http.(*Server).Serve in goroutine 99
	/usr/local/go/src/net/http/server.go:3454 +0x8ca
    client_test.go:112:
        	Error Trace:	/home/ubuntu/fleet-server/internal/pkg/es/client_test.go:112
        	Error:      	Received unexpected error:
        	            	EOF
        	Test:       	TestClientCerts/uses_certs
=== RUN   TestClientCerts/client_cert_does_not_match
2025/03/21 19:20:40 http: panic serving 127.0.0.1:58120: EVP_KDF_derive
openssl error(s):
error:1C800069:Provider routines::invalid key length
	../providers/implementations/kdfs/hkdf.c:163
goroutine 104 [running]:
net/http.(*conn).serve.func1()
	/usr/local/go/src/net/http/server.go:1947 +0x10a
panic({0x18c4820?, 0xc000036da0?})
	/usr/local/go/src/runtime/panic.go:787 +0x132
crypto/tls/internal/tls13.ExpandLabel[...](0xc00015f140, {0xc00002b040, 0x20, 0x20}, {0x1a29d2f, 0x2}, {0x0, 0x0, 0x0}, 0xc)
	/usr/local/go/src/crypto/tls/internal/tls13/tls13.go:41 +0x413
crypto/tls.(*cipherSuiteTLS13).trafficKey(0x25aacc0, {0xc00002b040, 0x20, 0x20})
	/usr/local/go/src/crypto/tls/key_schedule.go:29 +0x1bd
crypto/tls.(*halfConn).setTrafficSecret(0xc0001a2508, 0x25aacc0, 0x2, {0xc00002b040, 0x20, 0x20})
	/usr/local/go/src/crypto/tls/conn.go:234 +0x106
crypto/tls.(*serverHandshakeStateTLS13).sendServerParameters(0xc00015f610)
	/usr/local/go/src/crypto/tls/handshake_server_tls13.go:779 +0xa7e
crypto/tls.(*serverHandshakeStateTLS13).handshake(0xc00015f610)
	/usr/local/go/src/crypto/tls/handshake_server_tls13.go:80 +0xc5
crypto/tls.(*Conn).serverHandshake(0xc0001a2388, {0x1bed290, 0xc0003622d0})
	/usr/local/go/src/crypto/tls/handshake_server.go:56 +0x25d
crypto/tls.(*Conn).handshakeContext(0xc0001a2388, {0x1bed258, 0xc00018cb70})
	/usr/local/go/src/crypto/tls/conn.go:1568 +0x603
crypto/tls.(*Conn).HandshakeContext(...)
	/usr/local/go/src/crypto/tls/conn.go:1508
net/http.(*conn).serve(0xc0001b4360, {0x1bed258, 0xc00048e690})
	/usr/local/go/src/net/http/server.go:1971 +0x433
created by net/http.(*Server).Serve in goroutine 114
	/usr/local/go/src/net/http/server.go:3454 +0x8ca
--- FAIL: TestClientCerts (2.14s)
    --- FAIL: TestClientCerts/no_certs (0.36s)
    --- FAIL: TestClientCerts/uses_certs (0.81s)

[michel-laterman]

Note this seems to only occur when using the microsoft/go toolchain.

Digging into cert generation for a test; the test is configured with:

fleet-server/internal/pkg/es/client_test.go

Lines 69 to 87 in 2e51560

	ca := certs.GenCA(t)
	server := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("X-Elastic-Product", "Elasticsearch")
	fmt.Fprintln(w, "You know, For Search.")
	}))
	certPool := x509.NewCertPool()
	certPool.AddCert(ca.Leaf)
	// test server will verify a client cert if present
	server.TLS = &tls.Config{
	Certificates: []tls.Certificate{ca},
	ClientAuth: tls.VerifyClientCertIfGiven,
	ClientCAs: certPool,
	MinVersion: tls.VersionTLS12,
	}
	server.StartTLS()
	defer server.Close()
	cert := certs.GenCert(t, ca)

The error messages for keylength indicate the issue is with key length, we use 2048 (fips 140 2/3 requirement):

fleet-server/internal/pkg/testing/certs/certs.go

Line 93 in 2e51560

caKey, err := rsa.GenerateKey(rand.Reader, 2048) // less secure key for quicker testing.

I'm not sure what is causing these failures

[michel-laterman]

Looks like something similar has been reported and fixed: golang-fips/openssl#253

Manually patching in the changes from the PR associated with the fix: golang-fips/openssl#260 fixed all above test failures
could not recreate the original failure within the multipass vm, this does not appear to solve the failures on the test ami used to discover them

[michel-laterman]

It may be something that was patched within openssl.
The test AMI where this is occuring is running

OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)

However the multipass instance I tried to recreate within ran

OpenSSL 3.0.13 30 Jan 2024 (Library: OpenSSL 3.0.13 30 Jan 2024)

[cmacknz]

The source line doesn't match up with line 163 in the panic, but in the OpenSSL source it looks like this error gets hit when the key is empty.

https://github.com/openssl/openssl/blob/1bf649b5998ac98511203f54ce954eccfaf75467/providers/implementations/kdfs/hkdf.c#L167-L170

The code that has that error in that provider also hasn't changed between 3.0.2 and 3.0.13 so the problem must be somewhere else.

• https://github.com/openssl/openssl/blob/openssl-3.0.2/providers/implementations/kdfs/hkdf.c
• https://github.com/openssl/openssl/blob/openssl-3.0.2/providers/implementations/kdfs/hkdf.c

There are quite a lot of commits between the two releases so might be hard to narrow down what change caused this.

It might also be interesting to check if you get the same behavior with MS Go 1.23 vs MS Go 1.24.

[nicholasberlin]

I ran this on a VM from elasticcloud-fips-ubuntu-2204-amd64-dev-jammyf-250227183423 and it output this:

$ ./openssl-fips-test 
Checking OpenSSL lifecycle assurance.
*** Running check: FIPS module is available...
    SHA1 : (KAT_Digest) : Pass
    SHA2 : (KAT_Digest) : Pass
    SHA3 : (KAT_Digest) : Pass
    AES_GCM : (KAT_Cipher) : Pass
    AES_ECB_Decrypt : (KAT_Cipher) : Pass
    RSA : (KAT_Signature) :     RNG : (Continuous_RNG_Test) : Pass
    RNG : (Continuous_RNG_Test) : Pass
    RNG : (Continuous_RNG_Test) : Pass
Pass
    ECDSA : (KAT_Signature) : Pass
    ECDSA : (KAT_Signature) : Pass
    TLS13_KDF_EXTRACT : (KAT_KDF) : Pass
    TLS13_KDF_EXPAND : (KAT_KDF) : Pass
    TLS12_PRF : (KAT_KDF) : Pass
    PBKDF2 : (KAT_KDF) : Pass
    SSHKDF : (KAT_KDF) : Pass
    KBKDF : (KAT_KDF) : Pass
    HKDF : (KAT_KDF) : Pass
    SSKDF : (KAT_KDF) : Pass
    X963KDF : (KAT_KDF) : Pass
    X942KDF : (KAT_KDF) : Pass
    HASH : (DRBG) : Pass
    CTR : (DRBG) : Pass
    HMAC : (DRBG) : Pass
    DH : (KAT_KA) : Pass
    ECDH : (KAT_KA) : Pass
    HMAC : (Module_Integrity) : Pass
    Running check: FIPS module is available... passed.
*** Running check: EVP_default_properties_is_fips_enabled returns true... passed.
*** Running check: verify unapproved cryptographic routines are not available by default (e.g. MD5)... passed.

Lifecycle assurance satisfied.
Module details:
	name:     	Ubuntu 22.04 OpenSSL Cryptographic Module
	version:  	3.0.5-0ubuntu0.1+Fips2.1
	build:    	3.0.5-0ubuntu0.1+Fips2.1

Locate applicable CMVP certificates at
    https://csrc.nist.gov/projects/cryptographic-module-validation-program/validated-modules/search?SearchMode=Advanced&ModuleName=OpenSSL&CertificateStatus=Active&ValidationYear=0&SoftwareVersions=3.0.5$ ./openssl-fips-test 
Checking OpenSSL lifecycle assurance.
*** Running check: FIPS module is available...
    SHA1 : (KAT_Digest) : Pass
    SHA2 : (KAT_Digest) : Pass
    SHA3 : (KAT_Digest) : Pass
    AES_GCM : (KAT_Cipher) : Pass
    AES_ECB_Decrypt : (KAT_Cipher) : Pass
    RSA : (KAT_Signature) :     RNG : (Continuous_RNG_Test) : Pass
    RNG : (Continuous_RNG_Test) : Pass
    RNG : (Continuous_RNG_Test) : Pass
Pass
    ECDSA : (KAT_Signature) : Pass
    ECDSA : (KAT_Signature) : Pass
    TLS13_KDF_EXTRACT : (KAT_KDF) : Pass
    TLS13_KDF_EXPAND : (KAT_KDF) : Pass
    TLS12_PRF : (KAT_KDF) : Pass
    PBKDF2 : (KAT_KDF) : Pass
    SSHKDF : (KAT_KDF) : Pass
    KBKDF : (KAT_KDF) : Pass
    HKDF : (KAT_KDF) : Pass
    SSKDF : (KAT_KDF) : Pass
    X963KDF : (KAT_KDF) : Pass
    X942KDF : (KAT_KDF) : Pass
    HASH : (DRBG) : Pass
    CTR : (DRBG) : Pass
    HMAC : (DRBG) : Pass
    DH : (KAT_KA) : Pass
    ECDH : (KAT_KA) : Pass
    HMAC : (Module_Integrity) : Pass
    Running check: FIPS module is available... passed.
*** Running check: EVP_default_properties_is_fips_enabled returns true... passed.
*** Running check: verify unapproved cryptographic routines are not available by default (e.g. MD5)... passed.

Lifecycle assurance satisfied.
Module details:
	name:     	Ubuntu 22.04 OpenSSL Cryptographic Module
	version:  	3.0.5-0ubuntu0.1+Fips2.1
	build:    	3.0.5-0ubuntu0.1+Fips2.1

Locate applicable CMVP certificates at
    https://csrc.nist.gov/projects/cryptographic-module-validation-program/validated-modules/search?SearchMode=Advanced&ModuleName=OpenSSL&CertificateStatus=Active&ValidationYear=0&SoftwareVersions=3.0.5

I believe the source we are looking for is an ubuntu specific package that is hidden behind ubuntu pro subscription.

[michel-laterman]

TLS issue has been resolved upstream