Weird error in different situations: "Unable to exchange encryption keys"

[kalaksi]

Hi,

I have a weird problem with SSH handshake that I didn't have previously. Maybe related to SSH server getting updated at some point, I don't know.

The thing is that if I build a flatpak, it works, but if I do just e.g. cargo run it doesn't.
The error is "[Session(-1)] Unable to exchange encryption keys".

Cargo.lock says that ssh2-crate version is 0.9.5.
Regular OpenSSH client works.

Without flatpak, the SSH server logs show:

Connection from 10.0.20.10 port 43390 on 10.0.50.10 port 22 rdomain ""
debug1: Local version string SSH-2.0-OpenSSH_9.7
debug1: Remote protocol version 2.0, remote software version libssh2_1.11.1_DEV
debug1: compat_banner: no match: libssh2_1.11.1_DEV
debug2: fd 4 setting O_NONBLOCK
debug3: ssh_sandbox_init: preparing seccomp filter sandbox
debug2: Network child is on pid 212790
debug3: preauth child monitor started
debug3: privsep user:group 997:996 [preauth]
debug1: permanently_set_uid: 997/996 [preauth]
debug3: ssh_sandbox_child: setting PR_SET_NO_NEW_PRIVS [preauth]
debug3: ssh_sandbox_child: attaching seccomp filter program [preauth]
debug3: append_hostkey_type: ssh-rsa key not permitted by HostkeyAlgorithms [preauth]
debug1: list_hostkey_types: rsa-sha2-512,rsa-sha2-256,ssh-ed25519 [preauth]
debug3: send packet: type 20 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
*HANGS HERE*

And with flatpak:

Connection from 10.0.20.10 port 51588 on 10.0.50.10 port 22 rdomain ""
debug1: Local version string SSH-2.0-OpenSSH_9.7
debug1: Remote protocol version 2.0, remote software version libssh2_1.11.1_DEV
debug1: compat_banner: no match: libssh2_1.11.1_DEV
debug2: fd 4 setting O_NONBLOCK
debug3: ssh_sandbox_init: preparing seccomp filter sandbox
debug2: Network child is on pid 213553
debug3: preauth child monitor started
debug3: privsep user:group 997:996 [preauth]
debug1: permanently_set_uid: 997/996 [preauth]
debug3: ssh_sandbox_child: setting PR_SET_NO_NEW_PRIVS [preauth]
debug3: ssh_sandbox_child: attaching seccomp filter program [preauth]
debug3: append_hostkey_type: ssh-rsa key not permitted by HostkeyAlgorithms [preauth]
debug1: list_hostkey_types: rsa-sha2-512,rsa-sha2-256,ssh-ed25519 [preauth]
debug3: send packet: type 20 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug3: receive packet: type 20 [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug2: local server KEXINIT proposal [preauth]
etc...

The point where it hangs is where client-side error is returned from handshake():

        session_data.session = ssh2::Session::new().unwrap();
        session_data.session.set_tcp_stream(stream);
        if let Err(error) = session_data.session.handshake() {

Any idea what could be causing this?

[kalaksi]

Okay, I've finally narrowed it down to a manageable amount of code, but I still have no idea what the problem could be.

I have set up a new test project for only testing this.
Cargo.toml:

[package]
name = "weirdness"
version = "0.1.0"
edition = "2021"

[[bin]]
name = "test"
path = "src/main.rs"

[dependencies]
ssh2 = "0.9.5"
qmetaobject = "0.2.10"
# Excluding openssl makes it work again.
openssl = { version = "0.10.71", features = ["vendored"] }

src/main.rs:

use std::{net::TcpStream, thread::{self, sleep}};
use qmetaobject::QmlEngine;

fn main() {
    let handle = thread::spawn(move || {
        println!("Thread starting");

        // If sleeping here, QML gets loaded first and handshake fails.
        sleep(std::time::Duration::from_secs(1));

        let stream = TcpStream::connect("10.0.50.10:22").unwrap();
        println!("Connected");
        let mut session = ssh2::Session::new().unwrap();
        session.set_tcp_stream(stream);
        if let Err(error) = session.handshake() {
            println!("Error: {}", error);
        }
        else {
            println!("Success");
        }
    });

    let mut engine = QmlEngine::new();
    println!("Loading QML");
    // If commenting out load_data(), SSH handshake succeeds.
    engine.load_data(r#"import QtQuick.Controls

ApplicationWindow {
}"#.into());
    println!("QML loaded");
    handle.join().unwrap();
    println!("Exiting");
}

Note the comments. If I exclude openssl crate it starts to work again. If I don't sleep 1 sec in the thread (so it connects before loading QML) it starts to work again. And similarly, if I remove loading of QML completely it starts to work again.

[kalaksi]

Tried to use features = ["vendored-openssl"] with ssh2 too but no success there.
BUT I actually got the test project to work correctly now!

I'm not really sure what's going on but manually calling SslConnector::builder() as the first thing will set things straight. Like so:

...
extern crate openssl;
use openssl::ssl::SslConnector;

fn main() {
    let _ = SslConnector::builder(openssl::ssl::SslMethod::tls()).unwrap();
...

I'd still like to know what's going on and if there's a more correct solution or if I'm doing something wrong.