Configuration design: Separate Access Management?

[malex984]

Note: this is a separation of super-issue #13

Station: 4.5 Why do we need key_ref? If you configured passwordless ssh then you only need the username to connect. The correct key is picked by ssh.

Station: 4.6 Also, once again. I really don't understand why you'd ever publicize a private key and if I can't understand it and you can't explain it to me then you'll have a harder time explaining it to a customer or interested party. We shouldn't "invent" our own security mechanisms but instead use things already in place.

[porst17]

The ssh key has to be stored on the machine the management cli is running on. The cli user needs read access to to the private key anyway. Why is it more dangerous to put the ssh password or private ssh key (reference) into the config than just keeping it in ~/.ssh? If am not mistaken, the ui and cli processes need access to the key (even though it might be indirectly by calling the ssh command with same permissions that in turn reads the key).

[malex984]

Alex said:

4.5 Why do we need key_ref? If you configured passwordless ssh then
you only need the username to connect. The correct key is picked by ssh.

I assume that you refer to using ~/.ssh/config and ~/.ssh/id_rsa, right?
We will use them if they are available BUT they may be absent!
Therefore this is an OPTIONAL source for configuration of password-less ssh
to remote hosts that cannot be accessed via default ssh configuration on
the server.

• 4.6 Also, once again. I really don't understand why you'd ever
  publicize a private key and if I can't understand it and you can't explain
  it to me then you'll have a harder time explaining it to a customer or
  interested party. We shouldn't "invent" our own security mechanisms but
  instead use things already in place.

Sure no inventions here - we directly use SSH keys!

Just consider:

• how exactly completely empty server system gets those keys?
• who will configure password-less ssh access to ALL ~150 stations for us
  in advance?

Did you read the comments regarding ssh keys in the document Installation exhibition computers ESO/HITS/Imaginary?

[malex984]

Eric said:

I assume that you refer to using ~/.ssh/config and ~/.ssh/id_rsa, right?
We will use them if they are available BUT they may be absent!
Therefore this is an OPTIONAL source for configuration of password-less ssh to remote hosts that cannot be accessed via default ssh configuration on the server.

Explain the mechanism. The server has a private key and a public key, and there are no reasons why it wouldn't have them. The stations need the public key in its authorized hosts list, and this is transferred just once in a secure manner because it needs to be done using the station's password... so it makes no sense to have it in the cfg.

Is there some other mechanism? Because anyway nothing but a public key should leave a system... the private key shouldn't be transmitted or known by external systems.

Sure no inventions here - we directly use SSH keys!

Just consider:

• how exactly completely empty server system gets those keys?
• who will configure password-less ssh access to ALL ~150 stations for us in advance?

The same way they'll install an OS, Docker, setup networking, a user, backups, OMD, etc. on 150 stations... This is well beyond the scope of our tool! There are countless tools for parallel administration / deployment. Setting up passwordless ssh access is a security-sensitive operation that is done just once, so it has no reason to be in the config.

... But, how to set up passwordless ssh to 150 stations?

With a list of 150 hostnames + their passwords + a script that transfers the server's public key to the authorized_keys file in the home of the user that will run docker using ssh with the password. Then you delete the list of hostnames + passwords.

Though, once again, we don't need to take care of this with hilbert.

Did you even read the comments regarding ssh keys in the document "Installation exhibition computers ESO/HITS/Imaginary"?

I don't have access to that document I think (just searched)... but if the idea completely vulnerates the security of stations and ssh credentials then it doesn't matter.

If I'm failing to understand something and this is indeed reasonable and perfectly secure then consider a prospective customer might have the same concerns I have and go away, so we need to be clear and avoid misunderstandings.

[elondaits]

The ssh key has to be stored on the machine the management cli is running on. The cli user needs read access to to the private key anyway. Why is it more dangerous to put the ssh password or private ssh key (reference) into the config than just keeping it in ~/.ssh? If am not mistaken, the ui and cli processes need access to the key (even though it might be indirectly by calling the ssh command with same permissions that in turn reads the key).

Normal scheme: To connect to the station from the server you have both a private key and a public key in the server and have an authorized_keys file in the station with a copy of your public key.

Now, this is the relevant piece of cfg:

  supernova:
    rexec: !!Unix {}
    address: “supernova.mfo.de”
    ssh: { user: "ur", key_ref: "supernova.id_rsa" }

Problems with this:

• It's a key associated with the station. You shouldn't have that in the server, nor need it. You only need the server's key, which is the same for all stations.
• It's apparently (based on the name), the private key, which should never leave the machine so there's less reason to specify it because it's used internally by ssh.

Alex said in previous posts this was a scheme to install the ssh keys in the stations, which I don't think it's something our system should do, because it's not specific to it and its just one of many deployment tasks.

[porst17]

@elondaits, you didn't answer my question, I think. You said above that we should never publicize a private key. But if the private key or a reference to it is in the config, it is not more public than a private key that is stored in .ssh. The private key must also not be password protected. Otherwise, we can't automatically connect to the stations. Or the key passphrase for the private key has to be put into the config file. An encrypted private key together with its password is as insecure a unencrypted private key. And the hilbert system needs access to the config file as well as to .ssh (because hilbert calls ssh and therefore is the same user).

But I would agree, that we could just leave out the key or key_ref part. The stations need the public key be set in their authorized_keys file. And the management server will just have the private key in its .ssh folder. It has to be put there by the admin or by the external syncing script.

@malex984 Why would you need to put the private key into the config in order to set up password-less ssh? If password-less ssh is not configured yet, you need a password to log in. Then you place the public key into the authorized_keys files.

Conclusion: Our system relies on properly set up password-less ssh access. Setting this up is a sysadmins job.

[elondaits]

@porst17 I didn't answer it because I think we're thinking of different things... you are talking about the server's private key (which is in the same machine as this cfg), but I assume that the private key that is associated to a station in the cfg is in fact the private key of the station (because the private key of the server is the same for all the stations and because Alex said it was for deploying the ssh keys to multiple stations easily).

A reference (path) to the server's private key doesn't expose anything, of course... my issue was with putting references to the stations' private key files, which suggests they're accesible by the server.

As for putting the server's private key in the file, yes, that could be a security problem... because the private key should be non writable and only readable by the user that connects through ssh with the station... while the cfg file could be readable/writeable by a different user or group (specially if it's synced through djangoplicity).

[malex984]

It seems that for this project WE are going to generate a single ssh key pair and during the general SW installation phase our public ssh key will be added to authorized_keys on each station by others (automatically via puppet).

Let us consider the management back-end in more details:

• management backend will be running in a docker container, but its image should not contain any private ssh keys!
• it may be necessary to access the server host system via ssh from within the management container since server is to be treated just like a station (e.g. for updating its configs or turning it off). It may be easily done with a temporary ssh key pair (no need to specify this in the general config file).
• there must be no manual intervention by us or admins to the server host system since in case of server's failure - the 2nd server host must to get all the latest data & configurations from CMS automatically all by itself assuming only initial Hilbert's Server Configuration Package (.RPM) was installed.
• another ssh key pair may be required to access CMS (to get the general configuration file via scp / sftp).

The above means that private ssh keys may be installed as a part of the initial Hilbert's Server Configuration Package (which will also include all docker images and Hilbert-Station CLI with initial config for the server as a station).

AFAIK Management container may easily access private keys that where previously installed on the server in many ways.

But what about <SERVER_HOST>/<HILBERT_USER>/.ssh/config?
Should it be installed just like the private key - via some extra Server Access Configuration Package (.RPM) during during the general SW installation phase?
Then it has to know ALL possible stations and CMS + their ssh connection details right from the very beginning! Moreover in order to change or update .ssh/config IT stuff will have to recreate that that .RPM and re-install it on both servers :-(

On the other hand i propose to specify which of those pre-installed private ssh keys (if there are several) is required to access which station (with all ssh connection details) in the general configuration file. No pre-installed .ssh/config is required since it may be generated on-the-fly from the general configuration file if necessary.

In this case the initial Hilbert's Server Configuration Package will only pre-configure access to CMS for downloading the general configuration file (which is required in any case).

ps: in the above i assumed that our servers will be running exactly the same low-level Hilbert-CLI-Station scripts as any other Linux station but will be configured with a very special profile (server) to run some special services that require quite a few local settings.

NOTE in what follows i tried to describe my understanding of the general responsibilities:

• We prepare:
  1. public/private ssh key pairs
  2. SW packages (.RPMs) with
     • Hilbert-Station-Scripts for a generic linux station
     • Initial configuration for generic stations (maybe incl. a few service images)
     • Testing configuration for generic standalone interactive stations (incl. images for the standalone testing application)
     • Initial configuration for the server (including ALL docker images with service & applications)
• IT Admins will (automatically via their puppet):
  1. create special users for Hilbert and add our public ssh key into authorized_keys on all interactive stations (not sure about the server)
  2. install necessary SW packages on interactive stations and servers

NOTE: maybe the 1st item above may also be performed by one of our SW packages (i.e. be part of 2nd step).

• There must be no manual intervention by us or admins into any of stations and the server (either prior to that initial installation or afterwards).
• Upon Power-On ANY station with Hilbert has to automatically start low-level Hilbert-CLI-Station's script and proceed according to its local configuration.

NOTE: servers are stations with special configurations - they start: Registry, OMD Server, Management Dashboard etc.

Configuration Update procedure:

If triggered Management Dashboard should be able to:

• pull ALL the necessary docker images & general config file from CMS (or update them)
• create/update the server's registry with the new docker images
• (re-)configure OMD service to monitor ALL stations (according to the updated general config file)
• for each station:
  • (re-)generate station's local Hilbert-Configuration (in simple bash)
  • Power It On (if necessary according to the configuration)
  • deploy newly generated Station's configurations (during station's autostart pause)

[porst17]

@elondaits I think, I have a better understanding of your remarks now.

@malex984 This is again a lot of text. Do you have any specific questions?

If you don't want to expose private keys to the management backend container, we could ask for ssh-agent running on the server and then forward the agent into the container. (Does not work on docker+mac right now, though. See docker/for-mac#483 and docker/for-mac#410.)

If you really need ~/.ssh/config, you would need to copy/mount it from the host into the container.

Managing and deploying keys should not be our concern. This is the IT admins job. To us, it doesn't matter if the keys are set up in advance by hand, using puppet or self-tailored shell scripts. Also, replication to a second backup server is not our task.

The management of the ssh keys and ssh config can be part of the syncing. This should solve most of the issues you stated above. The sync script will be specific for each installation of hilbert. I don't indent to provide a generic solution for the syncing.

Hilbert has some requirements. Hilbert does not care, if the IT admin is fulfilling these requirements by hand, via puppet or via a hand-written bash-script that syncs via some external CMS.

Hilbert takes the host name/ip (address) specified in the config and connects via ssh. If the host, the management container is running on, is able to connect and log in to address, then hilbert must be able to do so as well. That's all.

[malex984]

@malex984 This is again a lot of text. Do you have any specific questions?

Sorry, i will try to be as brief as possible.

I think we are safe to assume that ssh key pairs for stations and access to CMS are fixed and installed separately ahead of time and they will probably never change later on, correct?

If you really need ~/.ssh/config,

Either

1. we rely on IT Admins to maintain ~/.ssh/config on all servers for us or
2. corresponding connection details go into our single general config file

NOTE:

• 1. means another configuration file for IT Admins to maintain
• 2. means that Hilbert-Server can automatically create .ssh/config if necessary

The management of the ssh keys and ssh config can be part of the syncing.

Do you mean Hilbert-Server getting them from CMS? This is how it is done in the current prototype. I thought that was not appropriate for you guys?

Hilbert does not care, if the IT admin is fulfilling these requirements by hand, via puppet or via a hand-written bash-script that syncs via some external CMS.

I considered the case of primary server crash: they power on a backup server - upon power-on it has to automatically download something from CMS and become as good as the failed one. It seems possible to me only if backup server has pre-configured access to up-to-date backup of all data & SW packages & configurations. Therefore either:

1. Admins regularly backup primary server or
2. All that data/SW/configs originates externally and can be used to automatically restore a server.

[elondaits]

I think we are safe to assume that ssh key pairs for stations and access to CMS are fixed and installed separately ahead of time and they will probably never change later on, correct?

When you say "ssh key pairs" it sounds like they're a lot. But it's just the private/public pair in the server. The public has to be copied to the authorized_keys of every station... just that. If the server accesses the CMS then it's the same private, with the public copied to the CMS. If the CMS accesses the server then it will need to install its own public key in the server... but that's something outside the scope of hilbert.

[malex984]

If the server accesses the CMS then it's the same private, with the public copied to the CMS.

AFAIR, CMS access was never discussed in this regard.

When you say "ssh key pairs" it sounds like they're a lot.

Yes, for the current project we may do with a single pair for accessing:

1. CMS and
2. all stations (including the server host system)

from within the container with the Hilbert management backend.

Note: that ssh pair will be installed on the server host system! I can see the following way to use it by management container:

1. mounting the place with ssh keys as a data volume

• the place with ssh keys has to be a part of server configuration (as a station). Which is fine as they will likely be installed simultaneously by the same .RPM.

2. using plain text environment variables

• as above, also
• may be cumbersome if it becomes necessary to handle several keys in general case

3. setting up ssh agent socket on the host and mounting it into the container

• requires ssh agent running on the host server system
• may not work if we use a different non-root user within the management container

Currently i tend to read-only mount ${HOME}/ into the management container (e.g. as /HOME).