Windows unexpected or badly formatted, MacOS bad protocol version

[fufesou]

version: 0.2.12

We're currently facing this issue rustdesk/rustdesk-server-pro#417

It's the same to seanmonstar/reqwest#2004 and seanmonstar/reqwest#2411

Sample code

use native_tls::TlsConnector;
use std::io::{Read, Write};
use std::net::TcpStream;

fn main() {
    let connector = TlsConnector::new().unwrap();

    let stream = TcpStream::connect("test.com:443").unwrap();
    let mut stream = connector.connect("test.com", stream).unwrap();

    stream.write_all(b"GET / HTTP/1.0\r\n\r\n").unwrap();
    let mut res = vec![];
    stream.read_to_end(&mut res).unwrap();
    println!("{}", String::from_utf8_lossy(&res));
}

Output

Trace log on Windows

called `Result::unwrap()` on an `Err` value: Failure(Os { code: -2146893018, kind: Uncategorized, message: "The message received was unexpected or badly formatted." })

Trace log on MacOS

called `Result::unwrap()` on an `Err` value: Failure(Error { code: -9836, message: "bad protocol version" })

Curl -vv

* Connected to test.com (54.36.110.163) port 443 (#0)
* ALPN: offers h2,http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
*  CAfile: D:/Program Files/Git/mingw64/etc/ssl/certs/ca-bundle.crt
*  CApath: none
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [15 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2838 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [520 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=test.com
*  start date: Oct 15 13:29:24 2024 GMT
*  expire date: Jan 13 13:29:23 2025 GMT
*  subjectAltName: host "test.com" matched cert's "*.test.com"
*  issuer: C=US; O=Let's Encrypt; CN=R10
*  SSL certificate verify ok.
} [5 bytes data]
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /api/oidc/auth]
* h2h3 [:scheme: https]
* h2h3 [:authority: test.com]
* h2h3 [user-agent: curl/7.88.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x1f41c451480)
} [5 bytes data]
> GET /api/oidc/auth HTTP/2
> Host: test.com
> user-agent: curl/7.88.1
> accept: */*
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [122 bytes data]
< HTTP/2 405
< allow: POST
< date: Wed, 16 Oct 2024 03:38:35 GMT
< content-length: 0

The most possible reason. rustdesk/rustdesk-server-pro#417 (comment)

[fufesou]

rustdesk/rustdesk-server-pro#417 (comment)

Hi, sorry to bother, but is there any progress?

[filips123]

My program has the same issue (initially repored in filips123/PWAsForFirefox#623).

I'm using reqwest as a HTTP client, and it works fine for basically all URLs, but fails specifically for https://www.midjourney.com/manifest.json (and probably other Midjourney URLs). I guess that you can use that URL for testing... I've also created a minimal example to reproduce this issue:

[dependencies]
anyhow = "1.0.95"
reqwest = { version = "0.12.12", features = ["blocking", "socks", "gzip", "brotli", "zstd", "deflate", "native-tls", "native-tls-alpn"] }

use anyhow::Result;
use reqwest::blocking::Client;
use reqwest::header::{HeaderMap, HeaderValue};

pub fn create_client() -> reqwest::Result<Client> {
    let mut headers = HeaderMap::new();
    headers.insert("Sec-Fetch-Site", HeaderValue::from_static("none"));
    headers.insert("Sec-Fetch-Dest", HeaderValue::from_static("manifest"));

    let builder = Client::builder()
        .user_agent("Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:100.0) Gecko/20100101 Firefox/100.0")
        .default_headers(headers);

    builder.build()
}

fn main() -> Result<()> {
    let url = "https://www.midjourney.com/manifest.json";

    let response = create_client()?
        .get(url.to_owned())
        .header(reqwest::header::REFERER, url.to_owned())
        .send()?
        .text()?;

    println!("{}", response);

    Ok(())
}

On Windows, it causes the following error:

Error: error sending request for url (https://www.midjourney.com/manifest.json)

Caused by:
    0: client error (Connect)
    1: The message received was unexpected or badly formatted. (os error -2146893018)

Meanwhile, curl works fine:

❯ curl -vvv -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:100.0) Gecko/20100101 Firefox/100.0" -H "Sec-Fetch-Site: none" -H "Sec-Fetch-Dest: manifest" --referer "https://www.midjourney.com/manifest.json" "https://www.midjourney.com/manifest.json"
* Host www.midjourney.com:443 was resolved.
* IPv6: (none)
* IPv4: 104.18.33.217, 172.64.154.39
*   Trying 104.18.33.217:443...
* Connected to www.midjourney.com (104.18.33.217) port 443
* schannel: disabled automatic use of client certificate
* using HTTP/1.x
> GET /manifest.json HTTP/1.1
> Host: www.midjourney.com
> User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:100.0) Gecko/20100101 Firefox/100.0
> Accept: */*
> Referer: https://www.midjourney.com/manifest.json
> Sec-Fetch-Site: none
> Sec-Fetch-Dest: manifest
>
* Request completely sent off
* schannel: remote party requests renegotiation
* schannel: renegotiating SSL/TLS connection
* schannel: SSL/TLS connection renegotiated
* schannel: failed to decrypt data, need more data
< HTTP/1.1 200 OK
< Date: Tue, 11 Feb 2025 19:40:33 GMT
< Content-Type: application/json; charset=UTF-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< x-xss-protection: 1; mode=block
< x-content-type-options: nosniff
< referrer-policy: origin-when-cross-origin
< x-frame-options: DENY
< content-security-policy: default-src 'self'; script-src 'self' https://challenges.cloudflare.com https://apis.google.com https://www.googletagmanager.com https://js.stripe.com/v3 https://www.googleadservices.com 'unsafe-inline' ; style-src 'self' https://fonts.googleapis.com 'unsafe-inline'; img-src * data: blob:; connect-src *; media-src 'self' https://updates.midjourney.com; frame-src 'self' https://authjourney.firebaseapp.com https://midjourney-web.firebaseapp.com https://player.vimeo.com https://localhost:3001 https://ideas.midjourney.com https://www.googletagmanager.com https://js.stripe.com https://editor.midjourney.com https://challenges.cloudflare.com; font-src 'self' https://fonts.gstatic.com; frame-ancestors 'self'; object-src 'none'; base-uri 'self'; form-action 'self';
< Cache-Control: public, max-age=0
< last-modified: Fri, 31 Jan 2025 00:49:55 GMT
< etag: W/"7a-194b9d5f738"
< vary: Accept-Encoding
< x-cloud-trace-context: XXXXXXXXXXXXXXXX
< via: 1.1 google
< strict-transport-security: max-age=31536000; includeSubDomains; preload
< alt-svc: h3=":443"; ma=86400
< cf-cache-status: DYNAMIC
< Set-Cookie: XXXXXXXXXXXXXXXX
< Server: cloudflare
< CF-RAY: XXXXXXXXXXXXXXXX-VIE
<
{
  "$schema": "https://json.schemastore.org/web-manifest-combined.json",
  "start_url": "/",
  "display": "standalone"
}
* Connection #0 to host www.midjourney.com left intact

Update while writing: According to this test, Midjourney only supports TLS 1.3. But rust-native-tls doesn't support it (#140). So this is probably the cause why this issue happens.