Problems with missing ssh config in Ansible when setting up Qubes staging environment

[rocodes]

Description

Problems when setting up Qubes staging environment, specifically SSH config (missing molecule-qubes-ssh-config), lead to inability to make staging.

Steps to Reproduce

• Qubes 4.0.3
• Create debian 10 standalone VM, follow docs to set up development dependencies (minus Virtualbox).
• Ensure VM is up to date
• Clone sd repo, activate virtualenv, pip install requirements
• Follow Qubes staging setup instructions
• Follow docs, except my dev VM is called dev instead of sd-dev (rpcpolicy include/admin-local-rwx and include/admin-global-rwx rules were adjusted accordingly)
• Set up base templates, Follow all server setup instructions including ssh-copy-id the key to each host, verify that can ssh from dev VM to each host with ssh sdadmin@10.137.0.5{0|1}
• Run make build-debs (success), then attempt to run make staging.

Expected Behavior

• make staging succeeds.

Actual Behavior

make staging fails in 2 ways, one resolvable, one that I'm stuck on.

TASK [Start Qubes VMs] *********************************************************
    failed: [localhost] (item={'name': 'app-staging', 'vm_base': 'sd-staging-app-base', 'vm_name': 'sd-staging-app', 'groups': ['securedrop_application_server', 'staging']}) => {"ansible_loop_var": "item", "changed": false, "cmd": ["qvm-start", "sd-staging-app"], "delta": "0:00:00.576945", "end": "2020-06-18 15:05:25.981141", "failed_when_result": true, "item": {"groups": ["securedrop_application_server", "staging"], "name": "app-staging", "vm_base": "sd-staging-app-base", "vm_name": "sd-staging-app"}, "msg": "non-zero return code", "rc": 1, "start": "2020-06-18 15:05:25.404196", "stderr": "Service call error: Request refused", "stderr_lines": ["Service call error: Request refused"], "stdout": "", "stdout_lines": []}
    failed: [localhost] (item={'name': 'mon-staging', 'vm_base': 'sd-staging-mon-base', 'vm_name': 'sd-staging-mon', 'groups': ['securedrop_monitor_server', 'staging']}) => {"ansible_loop_var": "item", "changed": false, "cmd": ["qvm-start", "sd-staging-mon"], "delta": "0:00:00.581352", "end": "2020-06-18 15:05:26.832194", "failed_when_result": true, "item": {"groups": ["securedrop_monitor_server", "staging"], "name": "mon-staging", "vm_base": "sd-staging-mon-base", "vm_name": "sd-staging-mon"}, "msg": "non-zero return code", "rc": 1, "start": "2020-06-18 15:05:26.250842", "stderr": "Service call error: Request refused", "stderr_lines": ["Service call error: Request refused"], "stdout": "", "stdout_lines": []}
    
    PLAY RECAP *********************************************************************
    localhost                  : ok=3    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0
    
ERROR: 
make: *** [Makefile:200: staging] Error 2

This "Service call error: Request refused" relates to this (resolved) Qubes issue and can be worked around.

Before workaround, a qvm- call from the dev vm to either sd-staging-app or sd-staging-mon gives us that 'Request refused' error:

$ qvm-prefs sd-staging-app
Traceback (most recent call last):
  File "/usr/bin/qvm-prefs", line 5, in <module>
    sys.exit(main())
  File "/usr/lib/python3/dist-packages/qubesadmin/tools/qvm_prefs.py", line 155, in main
    return process_actions(parser, args, target)
  File "/usr/lib/python3/dist-packages/qubesadmin/tools/qvm_prefs.py", line 96, in process_actions
    properties = target.property_list()
  File "/usr/lib/python3/dist-packages/qubesadmin/base.py", line 117, in property_list
    None)
  File "/usr/lib/python3/dist-packages/qubesadmin/base.py", line 68, in qubesd_call
    payload_stream)
  File "/usr/lib/python3/dist-packages/qubesadmin/app.py", line 688, in qubesd_call
    'Service call error: %s', stderr.decode())
qubesadmin.exc.QubesDaemonNoResponseError: Service call error: Request refused

Per a workaround, creating and tagging the VMs in dom0 solves this.

Run molecule create -s qubes-staging and see
Action: 'create': Skipping, instances already created.

Run molecule converge -s qubes-staging and see failure (output below of molecule --debug converge -s qubes-staging):

PLAY [Prepare servers for installation] ****************************************
    META: ran handlers
[WARNING]: raw module does not support the environment keyword
    <app-staging> ESTABLISH SSH CONNECTION FOR USER: None
    <app-staging> SSH: EXEC ssh -o ControlMaster=auto -o ControlPersist=1200 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=120 -o ControlPath=/home/user/.ansible/cp/e3c259c893 -tt app-staging 'sudo -H -S -n  -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-vylfealfiederupiqdenbvfouyxkdnzy ; apt -o Acquire::http::AllowRedirect=false update && apt -o Acquire::http::AllowRedirect=false --only-upgrade -y install apt'"'"''
[WARNING]: raw module does not support the environment keyword
    <mon-staging> ESTABLISH SSH CONNECTION FOR USER: None
    <mon-staging> SSH: EXEC ssh -o ControlMaster=auto -o ControlPersist=1200 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=120 -o ControlPath=/home/user/.ansible/cp/9d0957c823 -tt mon-staging 'sudo -H -S -n  -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-pogoebxyxxjuddzxcpioeppttwqpbyli ; apt -o Acquire::http::AllowRedirect=false update && apt -o Acquire::http::AllowRedirect=false --only-upgrade -y install apt'"'"''
    <app-staging> (255, b'', b'ssh_askpass: exec(/usr/bin/ssh-askpass): No such file or directory\r\nHost key verification failed.\r\n')
    
    TASK [prepare-servers : Ensure apt has been updated without following redirects] ***
    task path: /home/user/projects/securedrop/install_files/ansible-base/roles/prepare-servers/tasks/main.yml:8
    fatal: [app-staging]: UNREACHABLE! => {
        "changed": false,
        "msg": "Failed to connect to the host via ssh: ssh_askpass: exec(/usr/bin/ssh-askpass): No such file or directory\r\nHost key verification failed.",
        "unreachable": true
    }
    <mon-staging> (255, b'', b'ssh_askpass: exec(/usr/bin/ssh-askpass): No such file or directory\r\nHost key verification failed.\r\n')
    fatal: [mon-staging]: UNREACHABLE! => {
        "changed": false,
        "msg": "Failed to connect to the host via ssh: ssh_askpass: exec(/usr/bin/ssh-askpass): No such file or directory\r\nHost key verification failed.",
        "unreachable": true
    }
    
    NO MORE HOSTS LEFT *************************************************************
    
    NO MORE HOSTS LEFT *************************************************************
    
    PLAY RECAP *********************************************************************
    app-staging                : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0
    localhost                  : ok=3    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0
    mon-staging                : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0
    
ERROR: 

(Note the "ESTABLISH SSH CONNECTION FOR USER: None")

Comments

• I am missing /tmp/molecule-qubes-ssh-config (and possibly some other things that would facilitate ssh access via ansible to complete the staging env setup)
• Installing ssh-askpass in the dev VM is not the solution; a GUI appears prompting to add the host to known_hosts, while the ansible playbook errors out behind it.
• I do not see this error that conor mentioned, this is a slightly different issue

[rocodes]

@conorsch and @rmol are gods among men, this issue is resolved.

• For future selves, to spot qrexec errors, watch journalctl in dom0 via sudo journalctl -a -b -f | grep qrexec and then attempt to run playbook again in dev vm (and looking for 'deny' in the logs), although in this case there were errors
• Restart qubesd (note that I had rebooted my machine while trying to solve this, so this wasn't the first time qubesd was restarted, but...it helped?)
• From the dev vm, molecule destroy -s qubes-staging, molecule create -s qubes-staging (This was the first time that create was successful in the dev vm.)
• This time a Dom0 Operation execution dialog popped up that did not pop up before: operation admin.vm.device.mic.List with the source vm as my dev vm. I clicked Cancel (twice), but this was a good sign.
• The subsequent converge step completed successfully.

tl;dr: the dev vm has to be able to complete molecule create, and creating and tagging VMs in dom0 will only lead to suffering, even if molecule does not complain about that.