-
Notifications
You must be signed in to change notification settings - Fork 1.8k
Description
A note for the community
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Problem
I have vector configured on CentOS7 hosts sending logs to a central vector using a HTTP sink/source using TLS with client certificates.
The sending vectors log regular errors:
Apr 14 23:25:33 myhost vector[44196]: 2022-04-14T13:25:33.257534Z WARN sink{component_kind="sink" component_id=http component_type=http component_name=http}:request{request_id=4}: vector::sinks::util::retries: Retrying after error. error=Failed to make HTTP(S) request: error trying to connect: error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:ssl/record/rec_layer_s3.c:1544:SSL alert number 80
From https://datatracker.ietf.org/doc/html/rfc5246#appendix-A.3, alert number 80 is an internal error.
Both sender and receiver vector versions are:
vector 0.20.0 (x86_64-unknown-linux-gnu 2a706a3 2022-02-11)
Both sender and receiver hosts are CentOS7, with OpenSSL 1.0.2k-fips 26 Jan 2017
### Configuration
```text
---sender---
type = "http"
inputs = [
"journald",
]
compression = "gzip"
encoding = "ndjson"
tls.ca_file = "/etc/ssl/certs/ca.pem"
tls.crt_file = "/etc/ssl/certs/myhost.pem"
tls.enabled = true
tls.key_file = "/etc/ssl/private_keys/myhost.pem"
tls.verify_certificate = true
uri = "https://receiver:501"
---receiver---
type = "http"
address = "10.0.200.15:501"
framing.method = "newline_delimited"
tls.ca_file = "/etc/ssl/certs/ca.pem"
tls.crt_file = "/etc/ssl/certs/receiver.pem"
tls.enabled = true
tls.key_file = "/etc/ssl/private_keys/receiver.pem"
tls.verify_certificate = true
Version
vector 0.20.0 (x86_64-unknown-linux-gnu 2a706a3 2022-02-11)
Debug Output
2022-04-14T14:25:32.600128Z TRACE vector: Beep.
2022-04-14T14:25:33.254121Z TRACE sink{component_kind="sink" component_id=http component_type=http component_name=http}: vector::sinks::util::sink: Service ready; Sending batch.
2022-04-14T14:25:33.254187Z TRACE sink{component_kind="sink" component_id=http component_type=http component_name=http}: vector::sinks::util::sink: Submitting service request. in_flight_requests=1
2022-04-14T14:25:33.255217Z DEBUG sink{component_kind="sink" component_id=http component_type=http component_name=http}:request{request_id=2}:http: vector::internal_events::http_client: Sending HTTP request. uri=https://receiver.example.com:501/ method=POST version=HTTP/1.1 headers={"content-type": "application/x-ndjson", "content-encoding": "gzip", "user-agent": "Vector/0.20.0 (x86_64-unknown-linux-gnu 2a706a3 2022-02-11)", "accept-encoding": "identity"} body=[1728 bytes]
2022-04-14T14:25:33.256303Z DEBUG sink{component_kind="sink" component_id=http component_type=http component_name=http}:request{request_id=2}:http: vector::internal_events::http_client: HTTP error. error=error trying to connect: error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:ssl/record/rec_layer_s3.c:1544:SSL alert number 80
2022-04-14T14:25:33.256367Z WARN sink{component_kind="sink" component_id=http component_type=http component_name=http}:request{request_id=2}: vector::sinks::util::retries: Retrying after error. error=Failed to make HTTP(S) request: error trying to connect: error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:ssl/record/rec_layer_s3.c:1544:SSL alert number 80
2022-04-14T14:25:33.256385Z DEBUG sink{component_kind="sink" component_id=http component_type=http component_name=http}:request{request_id=2}: vector::sinks::util::retries: Retrying request. delay_ms=1000
2022-04-14T14:25:33.599960Z TRACE vector: Beep.
2022-04-14T14:25:34.258814Z DEBUG sink{component_kind="sink" component_id=http component_type=http component_name=http}:request{request_id=2}:http: vector::internal_events::http_client: Sending HTTP request. uri=https://receiver.example.com:501/ method=POST version=HTTP/1.1 headers={"content-type": "application/x-ndjson", "content-encoding": "gzip", "user-agent": "Vector/0.20.0 (x86_64-unknown-linux-gnu 2a706a3 2022-02-11)", "accept-encoding": "identity"} body=[1728 bytes]
2022-04-14T14:25:34.289470Z DEBUG sink{component_kind="sink" component_id=http component_type=http component_name=http}:request{request_id=2}:http: vector::internal_events::http_client: HTTP response. status=200 OK version=HTTP/1.1 headers={"content-length": "0", "date": "Thu, 14 Apr 2022 14:25:33 GMT"} body=[empty]
2022-04-14T14:25:34.289546Z TRACE sink{component_kind="sink" component_id=http component_type=http component_name=http}:request{request_id=2}: vector::internal_events::common: Bytes sent. byte_size=1728 protocol=https endpoint=https://receiver.example.com:501/
2022-04-14T14:25:34.289573Z TRACE sink{component_kind="sink" component_id=http component_type=http component_name=http}:request{request_id=2}: vector::internal_events::adaptive_concurrency: Changed concurrency. concurrency=1 reached_limit=true had_back_pressure=false current_rtt=Some(1.035237648s) past_rtt=231.674877ms past_rtt_deviation=246.289787ms
2022-04-14T14:25:34.289604Z TRACE sink{component_kind="sink" component_id=http component_type=http component_name=http}:request{request_id=2}: vector::sinks::util::sink: Response successful. response=Response { status: 200, version: HTTP/1.1, headers: {"content-length": "0", "date": "Thu, 14 Apr 2022 14:25:33 GMT"}, body: b"" }
2022-04-14T14:25:34.289623Z TRACE sink{component_kind="sink" component_id=http component_type=http component_name=http}:request{request_id=2}: vector_common::internal_event::events_sent: Events sent. count=16 byte_size=43208
2022-04-14T14:25:34.289654Z TRACE sink{component_kind="sink" component_id=http component_type=http component_name=http}: vector::sinks::util::sink: Acking events. acking_num=16
2022-04-14T14:25:34.599514Z TRACE vector: Beep.
2022-04-14T14:25:35.599929Z TRACE vector: Beep.
Example Data
Apr 14 23:25:33 myhost vector[44196]: 2022-04-14T13:25:33.257534Z WARN sink{component_kind="sink" component_id=http component_type=http component_name=http}:request{request_id=4}: vector::sinks::util::retries: Retrying after error. error=Failed to make HTTP(S) request: error trying to connect: error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:ssl/record/rec_layer_s3.c:1544:SSL alert number 80
Apr 14 23:35:29 myhost vector[44196]: 2022-04-14T13:35:29.419515Z WARN sink{component_kind="sink" component_id=http component_type=http component_name=http}:request{request_id=5}: vector::sinks::util::retries: Retrying after error. error=Failed to make HTTP(S) request: error trying to connect: error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:ssl/record/rec_layer_s3.c:1544:SSL alert number 80
Apr 14 23:40:29 myhost vector[44196]: 2022-04-14T13:40:29.429999Z WARN sink{component_kind="sink" component_id=http component_type=http component_name=http}:request{request_id=7}: vector::sinks::util::retries: Retrying after error. error=Failed to make HTTP(S) request: error trying to connect: error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:ssl/record/rec_layer_s3.c:1544:SSL alert number 80
Apr 14 23:45:29 myhost vector[44196]: 2022-04-14T13:45:29.587987Z WARN sink{component_kind="sink" component_id=http component_type=http component_name=http}:request{request_id=8}: vector::sinks::util::retries: Retrying after error. error=Failed to make HTTP(S) request: error trying to connect: error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:ssl/record/rec_layer_s3.c:1544:SSL alert number 80
Apr 14 23:53:02 myhost vector[44196]: 2022-04-14T13:53:02.398311Z WARN sink{component_kind="sink" component_id=http component_type=http component_name=http}:request{request_id=9}: vector::sinks::util::retries: Retrying after error. error=Failed to make HTTP(S) request: error trying to connect: error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:ssl/record/rec_layer_s3.c:1544:SSL alert number 80
Additional Context
The receiver's certificate has multiple Subject Alternative Names, one of which is the hostname used by the senders. The Subject: CN=other.example.com
, where the sender vector agents use receiver.example.com
as the HTTP sink hostname.
References
No response