Skip to content

HTTPS sink SSL Alert number 80 #12212

@gsalisbury

Description

@gsalisbury

A note for the community

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

I have vector configured on CentOS7 hosts sending logs to a central vector using a HTTP sink/source using TLS with client certificates.

The sending vectors log regular errors:

Apr 14 23:25:33 myhost vector[44196]: 2022-04-14T13:25:33.257534Z  WARN sink{component_kind="sink" component_id=http component_type=http component_name=http}:request{request_id=4}: vector::sinks::util::retries: Retrying after error. error=Failed to make HTTP(S) request: error trying to connect: error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:ssl/record/rec_layer_s3.c:1544:SSL alert number 80
From https://datatracker.ietf.org/doc/html/rfc5246#appendix-A.3, alert number 80 is an internal error.

Both sender and receiver vector versions are:
vector 0.20.0 (x86_64-unknown-linux-gnu 2a706a3 2022-02-11)
Both sender and receiver hosts are CentOS7, with OpenSSL 1.0.2k-fips  26 Jan 2017


### Configuration

```text
---sender---
type = "http"
inputs = [
  "journald",
]
compression = "gzip"
encoding = "ndjson"
tls.ca_file = "/etc/ssl/certs/ca.pem"
tls.crt_file = "/etc/ssl/certs/myhost.pem"
tls.enabled = true
tls.key_file = "/etc/ssl/private_keys/myhost.pem"
tls.verify_certificate = true
uri = "https://receiver:501"

---receiver---
type = "http"
address = "10.0.200.15:501"
framing.method = "newline_delimited"
tls.ca_file = "/etc/ssl/certs/ca.pem"
tls.crt_file = "/etc/ssl/certs/receiver.pem"
tls.enabled = true
tls.key_file = "/etc/ssl/private_keys/receiver.pem"
tls.verify_certificate = true

Version

vector 0.20.0 (x86_64-unknown-linux-gnu 2a706a3 2022-02-11)

Debug Output

2022-04-14T14:25:32.600128Z TRACE vector: Beep.
2022-04-14T14:25:33.254121Z TRACE sink{component_kind="sink" component_id=http component_type=http component_name=http}: vector::sinks::util::sink: Service ready; Sending batch.
2022-04-14T14:25:33.254187Z TRACE sink{component_kind="sink" component_id=http component_type=http component_name=http}: vector::sinks::util::sink: Submitting service request. in_flight_requests=1
2022-04-14T14:25:33.255217Z DEBUG sink{component_kind="sink" component_id=http component_type=http component_name=http}:request{request_id=2}:http: vector::internal_events::http_client: Sending HTTP request. uri=https://receiver.example.com:501/ method=POST version=HTTP/1.1 headers={"content-type": "application/x-ndjson", "content-encoding": "gzip", "user-agent": "Vector/0.20.0 (x86_64-unknown-linux-gnu 2a706a3 2022-02-11)", "accept-encoding": "identity"} body=[1728 bytes]
2022-04-14T14:25:33.256303Z DEBUG sink{component_kind="sink" component_id=http component_type=http component_name=http}:request{request_id=2}:http: vector::internal_events::http_client: HTTP error. error=error trying to connect: error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:ssl/record/rec_layer_s3.c:1544:SSL alert number 80
2022-04-14T14:25:33.256367Z  WARN sink{component_kind="sink" component_id=http component_type=http component_name=http}:request{request_id=2}: vector::sinks::util::retries: Retrying after error. error=Failed to make HTTP(S) request: error trying to connect: error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:ssl/record/rec_layer_s3.c:1544:SSL alert number 80
2022-04-14T14:25:33.256385Z DEBUG sink{component_kind="sink" component_id=http component_type=http component_name=http}:request{request_id=2}: vector::sinks::util::retries: Retrying request. delay_ms=1000
2022-04-14T14:25:33.599960Z TRACE vector: Beep.
2022-04-14T14:25:34.258814Z DEBUG sink{component_kind="sink" component_id=http component_type=http component_name=http}:request{request_id=2}:http: vector::internal_events::http_client: Sending HTTP request. uri=https://receiver.example.com:501/ method=POST version=HTTP/1.1 headers={"content-type": "application/x-ndjson", "content-encoding": "gzip", "user-agent": "Vector/0.20.0 (x86_64-unknown-linux-gnu 2a706a3 2022-02-11)", "accept-encoding": "identity"} body=[1728 bytes]
2022-04-14T14:25:34.289470Z DEBUG sink{component_kind="sink" component_id=http component_type=http component_name=http}:request{request_id=2}:http: vector::internal_events::http_client: HTTP response. status=200 OK version=HTTP/1.1 headers={"content-length": "0", "date": "Thu, 14 Apr 2022 14:25:33 GMT"} body=[empty]
2022-04-14T14:25:34.289546Z TRACE sink{component_kind="sink" component_id=http component_type=http component_name=http}:request{request_id=2}: vector::internal_events::common: Bytes sent. byte_size=1728 protocol=https endpoint=https://receiver.example.com:501/
2022-04-14T14:25:34.289573Z TRACE sink{component_kind="sink" component_id=http component_type=http component_name=http}:request{request_id=2}: vector::internal_events::adaptive_concurrency: Changed concurrency. concurrency=1 reached_limit=true had_back_pressure=false current_rtt=Some(1.035237648s) past_rtt=231.674877ms past_rtt_deviation=246.289787ms
2022-04-14T14:25:34.289604Z TRACE sink{component_kind="sink" component_id=http component_type=http component_name=http}:request{request_id=2}: vector::sinks::util::sink: Response successful. response=Response { status: 200, version: HTTP/1.1, headers: {"content-length": "0", "date": "Thu, 14 Apr 2022 14:25:33 GMT"}, body: b"" }
2022-04-14T14:25:34.289623Z TRACE sink{component_kind="sink" component_id=http component_type=http component_name=http}:request{request_id=2}: vector_common::internal_event::events_sent: Events sent. count=16 byte_size=43208
2022-04-14T14:25:34.289654Z TRACE sink{component_kind="sink" component_id=http component_type=http component_name=http}: vector::sinks::util::sink: Acking events. acking_num=16
2022-04-14T14:25:34.599514Z TRACE vector: Beep.
2022-04-14T14:25:35.599929Z TRACE vector: Beep.

Example Data

Apr 14 23:25:33 myhost vector[44196]: 2022-04-14T13:25:33.257534Z WARN sink{component_kind="sink" component_id=http component_type=http component_name=http}:request{request_id=4}: vector::sinks::util::retries: Retrying after error. error=Failed to make HTTP(S) request: error trying to connect: error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:ssl/record/rec_layer_s3.c:1544:SSL alert number 80
Apr 14 23:35:29 myhost vector[44196]: 2022-04-14T13:35:29.419515Z WARN sink{component_kind="sink" component_id=http component_type=http component_name=http}:request{request_id=5}: vector::sinks::util::retries: Retrying after error. error=Failed to make HTTP(S) request: error trying to connect: error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:ssl/record/rec_layer_s3.c:1544:SSL alert number 80
Apr 14 23:40:29 myhost vector[44196]: 2022-04-14T13:40:29.429999Z WARN sink{component_kind="sink" component_id=http component_type=http component_name=http}:request{request_id=7}: vector::sinks::util::retries: Retrying after error. error=Failed to make HTTP(S) request: error trying to connect: error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:ssl/record/rec_layer_s3.c:1544:SSL alert number 80
Apr 14 23:45:29 myhost vector[44196]: 2022-04-14T13:45:29.587987Z WARN sink{component_kind="sink" component_id=http component_type=http component_name=http}:request{request_id=8}: vector::sinks::util::retries: Retrying after error. error=Failed to make HTTP(S) request: error trying to connect: error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:ssl/record/rec_layer_s3.c:1544:SSL alert number 80
Apr 14 23:53:02 myhost vector[44196]: 2022-04-14T13:53:02.398311Z WARN sink{component_kind="sink" component_id=http component_type=http component_name=http}:request{request_id=9}: vector::sinks::util::retries: Retrying after error. error=Failed to make HTTP(S) request: error trying to connect: error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:ssl/record/rec_layer_s3.c:1544:SSL alert number 80

Additional Context

The receiver's certificate has multiple Subject Alternative Names, one of which is the hostname used by the senders. The Subject: CN=other.example.com, where the sender vector agents use receiver.example.com as the HTTP sink hostname.

References

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    sink: httpAnything `http` sink relatedsource: http_serverAnything `http_server` source relatedtype: bugA code related bug.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions