[RFE] Installer does not use existing SSH agent

[bison]

Version

$ ./current/openshift-install version
./current/openshift-install unreleased-master-1130-g311a8a1266268aa454e45049e85f2b3c186715c7-dirty
built from commit 311a8a1266268aa454e45049e85f2b3c186715c7
release image registry.svc.ci.openshift.org/ocp/release@sha256:e898c929fea0b02146b074ca848e012de243d41f90cf87fc2f5dca3455563ce2

Platform (aws|libvirt|openstack):

AWS

What happened?

The install failed, and the installer was not able to SSH to the machines to gather logs:

INFO Pulling debug logs from the bootstrap machine
ERROR failed to create SSH client: failed to initialize the SSH agent: [failed to parse SSH private key from "[REDACTED]": ssh: no key found, failed to parse SSH private key from "[REDACTED]": ssh: no key found, failed to parse SSH private key from "[REDACTED]": ssh: cannot decode encrypted private keys, failed to parse SSH private key from "[REDACTED]": ssh: no key found, failed to parse SSH private key from "[REDACTED]": ssh: no key found, failed to parse SSH private key from "[REDACTED]": ssh: no key found, failed to read "[REDACTED]: no such device or address]
FATAL waiting for Kubernetes API: context deadline exceeded

What you expected to happen?

The installer should have been able to gather the logs using my already running SSH agent. My private keys are not stored on disk. If the SSH connection could not be established, the installer should have provided me the commands to try manually.

How to reproduce it (as minimally and precisely as possible)?

Start with a machine with no SSH private keys available in ~/.ssh.

$ openshift-install create cluster

Anything else we need to know?

I think the support for gathering diagnostics via SSH introduced in #1822 could be a bit more robust in some cases. A few things:

• It looks like defaultPrivateSSHKeys() tries to load every file in ${HOME}/.ssh and parse them as an SSH private key. Maybe I'm reading wrong, but it looks like if any file fails to parse (e.g. a config or known_hosts file) LoadPrivateSSHKeys() returns a non-nil error which causes newAgent() to fail.

• Some users don't have private keys on disk. It's common for private keys to live on a smart-card and be accessed via an already running SSH agent. It would be nice if the installer detected SSH_AUTH_SOCK and forwarded to the existing agent.

• If all else fails, I think the installer should print the commands like it used to. That way users can try for themselves with their existing configs.

References

#1822

[bison]

Ah, I think I was reading defaultPrivateSSHKeys() wrong. Sorry about that. I think the points about existing SSH agents and printing manual instructions still stand though.

[bison]

I actually have a patch for this, but now my installs are succeeding and I'm not sure how I can test it end-to-end with the installer. Any tips @abhinavdahiya?

[abhinavdahiya]

I actually have a patch for this, but now my installs are succeeding and I'm not sure how I can test it end-to-end with the installer. Any tips @abhinavdahiya?

you can Ctrl-C after the API is up to prevent installer to remove the bootstrap node. and then use gather bootstrap subcommand.

you can also put an expired pull-secret for installation that will fail during bootstrap and trigger the gather bootstrap

[abhinavdahiya]

It looks like defaultPrivateSSHKeys() tries to load every file in ${HOME}/.ssh and parse them as an SSH private key. Maybe I'm reading wrong, but it looks like if any file fails to parse (e.g. a config or known_hosts file) LoadPrivateSSHKeys() returns a non-nil error which causes newAgent() to fail.

NO
if keys were specified, newAgent fails if any key fails to load.
if no keys were specified, newAgent fails only when defaultPrivateSSHKeys returns error, which is when no key could be loaded from ~/.ssh

If all else fails, I think the installer should print the commands like it used to. That way users can try for themselves with their existing configs.

I do not agree with this. Installer allows user to either give it the keys in the gather bootstrap or it tries to load the keys from user''s environment. There is not need to return the SSH command, as the installer is completely capable of running the SSH commands as long as it has the right keys...

And as for your use-case that you want the installer to use the SSH_AGENT already setup, I could get behind that as fall back if no keys were provided.

[bison]

Yeah, sorry I had misread the defaultPrivateSSHKeys bit. I don't really care too much about printing the commands, so no real argument from me on that point.

I think using an existing SSH agent is important though. I'll try your suggestion of just interrupting the installer after the API is up and open a PR if things work. Thanks!