Vector does not put the Proxy-Authorization header on the wire

[syedriko]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

With an HTTP proxy with a username and password configured for an http or an elasticsearch sink, vector does communicate with the configured proxy, but it does not include the Proxy-Authorization header in the outgoing HTTP request. Confirmed with Wireshark:

Hypertext Transfer Protocol
    GET http://foobar:9200/ HTTP/1.1\r\n
        [Expert Info (Chat/Sequence): GET http://foobar:9200/ HTTP/1.1\r\n]
            [GET http://foobar:9200/ HTTP/1.1\r\n]
            [Severity level: Chat]
            [Group: Sequence]
        Request Method: GET
        Request URI: http://foobar:9200/
        Request Version: HTTP/1.1
    user-agent: Vector/0.30.0-custom-5d3f619ef3 (x86_64-unknown-linux-gnu debug=full)\r\n
    accept-encoding: identity\r\n
    host: foobar:9200\r\n
    \r\n
    [Full request URI: http://foobar:9200/]
    [HTTP request 1/3]
    [Response in frame: 73]
    [Next request in frame: 100]

Configuration

data_dir = "/tmp"

[sources.log_generator]
type = "demo_logs"
format = "json"

[sinks.console]
inputs = ["log_generator"]
type = "console"
target = "stdout"
[sinks.console.encoding]
codec = "json"

[sinks.default]
type = "http"
inputs = ["log_generator"]
uri = "http://foobar:9200"
encoding.codec = "json"
proxy.http = "http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@localhost:9200"

Version

vector 0.30.0-custom-5d3f619ef3 (x86_64-unknown-linux-gnu debug=full)

Debug Output

No response

Example Data

No response

Additional Context

With inspiration from https://github.com/tafia/hyper-proxy/blob/aca1e1f5546ed5530e1a72dc5e07c5ed4b6099b3/src/lib.rs#L32, I hacked addition of proxy headers to the outgoing HTTP requests syedriko@27459d0 and got the Authorization and Proxy-Authorization headers added to the HTTP requests. Here's a capture from Wireshark:

Hypertext Transfer Protocol
    GET http://foobar:9200/ HTTP/1.1\r\n
        [Expert Info (Chat/Sequence): GET http://foobar:9200/ HTTP/1.1\r\n]
            [GET http://foobar:9200/ HTTP/1.1\r\n]
            [Severity level: Chat]
            [Group: Sequence]
        Request Method: GET
        Request URI: http://foobar:9200/
        Request Version: HTTP/1.1
    user-agent: Vector/0.30.0-custom-5d3f619ef3 (x86_64-unknown-linux-gnu debug=full)\r\n
    accept-encoding: identity\r\n
    authorization: Basic cHJveHktdXNlcjE6SllnVThxUlpWNERZNFBYSmJ4Sks=\r\n
        Credentials: proxy-user1:JYgU8qRZV4DY4PXJbxJK
    proxy-authorization: Basic cHJveHktdXNlcjE6SllnVThxUlpWNERZNFBYSmJ4Sks=\r\n
        Credentials: proxy-user1:JYgU8qRZV4DY4PXJbxJK
    host: foobar:9200\r\n
    \r\n
    [Full request URI: http://foobar:9200/]
    [HTTP request 1/4]
    [Response in frame: 17]
    [Next request in frame: 24]

Also the tests at

vector/lib/vector-core/src/config/proxy.rs

Line 345 in 5d3f619

first.headers().get("authorization"),

and below
appear to be the only place in the vector codebase where hyper_proxy::Proxy:headers() https://github.com/tafia/hyper-proxy/blob/aca1e1f5546ed5530e1a72dc5e07c5ed4b6099b3/src/lib.rs#L228 is called from, even though these are the headers that are populated at Proxy construction time.

References

No response

[zamazan4ik]

Would you like to contribute your changes to the upstream? :)

[syedriko]

Would you like to contribute your changes to the upstream? :)

I meant my changes more as a conversation starter :) Do you think this is the right way to fix this?

[spencergilbert]

Thanks for the report, and we'd be happy to accept a contribution fixing. I am curious about your first request you shared - it doesn't contain the Authorization header, which should be present? Especially given the link to tests asserting it gets created.

Am I missing something obvious there?

[syedriko]

Thanks for the report, and we'd be happy to accept a contribution fixing. I am curious about your first request you shared - it doesn't contain the Authorization header, which should be present? Especially given the link to tests asserting it gets created.

Am I missing something obvious there?

Here's my take: the build_proxy_with_basic_authorization() and build_proxy_with_special_chars_url_encoded() tests in

vector/lib/vector-core/src/config/proxy.rs

Line 323 in 5d3f619

fn build_proxy_with_basic_authorization() {

, being unit tests, check that the Authorization header gets created in the Proxy object. It is probably a good idea to check the Proxy-Authorization header as well, but that's not the main point.

These unit tests show that the needed header gets set, as a map entry, but the HTTP code never looks at those headers and never puts them on the wire, as no code besides these unit tests calls Proxy::headers(). And from https://github.com/tafia/hyper-proxy/blob/aca1e1f5546ed5530e1a72dc5e07c5ed4b6099b3/src/lib.rs#L32 it looks like the hyper-proxy crate expects the code using it to add proxy headers to the outgoing HTTP requests "manually". That's what my hack added.

Both Authorization and Proxy-Authorization headers get set in https://github.com/tafia/hyper-proxy/blob/aca1e1f5546ed5530e1a72dc5e07c5ed4b6099b3/src/lib.rs#L204, together. That's why there was neither of them in my first Wireshark capture.

[spencergilbert]

Ah, got it - misunderstood your message originally. Thanks for clarifying 🙏

[zamazan4ik]

Do you think this is the right way to fix this?

As far as I understand (but I could be wrong here) when the proxy is used - Proxy-Authorization header should (or must) be used, so your way is completely right. My only point - what if we have different basic auth credentials in the original and proxy connection strings? In this case, the values of Authorization and Proxy-Authorization shall be different.

[syedriko]

Do you think this is the right way to fix this?

As far as I understand (but I could be wrong here) when the proxy is used - Proxy-Authorization header should (or must) be used, so your way is completely right. My only point - what if we have different basic auth credentials in the original and proxy connection strings? In this case, the values of Authorization and Proxy-Authorization shall be different.

That ^^^ is sure something to keep in mind, but then there's the angle of how to code this correctly with the APIs of the Rust crates involved. I made a copy of the ProxyConnector for the HttpClient so I could access it in HttpClient::send. This is not looking like "the right way" at all. I hoped there'd be an interceptor where ProxyConnector could inject its headers.

[syedriko]

I've looked at the APIs of hyper, hyper-proxy and the vector http code and I'm not really seeing a better way of doing this, so here is a PR. @zamazan4ik I made setting of the proxy headers conditional, so if something sets [Proxy-]Authorization header prior, the new code will keep it.