Update admin and journalist workstations to use Tails 6

[zenmonkeykstop]

Tails 6 is in RC mode with a projected release date of Feb 27 2024. This version is based on Debian Bookworm and has a tonne of changes both under the hood and to the UX.

Updates to Tails 6 in terms of code changes and dependencies seem to be compatible with Tails 5 as well, so users still on Tails 5 can update to a version with said changes and upgrade tails independently (tho they will need to run setup again to remove the Python 3.9 virtualenv and set up a 3.11 one).

Update tasks roughly break out as follows:

code

• Update admin scripts for Tails 6
  • update bootstrap script to account for changes in apt's behaviour and Python version
  • update Python dependencies (stretch: upgrade to latest Ansible ✔️ )
  • test and update Ansible playbooks as needed
• Update network hook script (changes may be required if TCA or Tor itself behave differently)
• Update GUI updater for Tails 6 no changes necessary
• Update Gnome menu for Tails 6 no changes necessary

docs

• Update user documentation with any changes (for the sake of consistency any screenshots etc should be in existing light mode theme)

comms and release

• Plan comms - Tails users should be notified of requirement a
  decent time ahead of release. Users will probably have to do a CLI update after upgrading to Tails 6 as well.
• consider a blog post describing changes, a bit more substantial than usual release notification.
• schedule 2.8.0 release to coincide with or immediately follow Tails 6 release. At a minimum the Tails 6 changes should be in place in time

[zenmonkeykstop]

One thing I'm seeing immediately: git clones over https of the repo fail consistently for me with "unexpected disconnect" errors during object syncing. This could just be my slow connection + Tor stuff, but it's new to me. (Adding a --depth 1 reduced the overall download and got it working again.)

[zenmonkeykstop]

(WIP)

Working on the branch tails6-ftw, only changes so far are the admin dockerfile bump to bookworm, some deps changes, and the OS/stale virtualenv check.

• need to run sudo apt-get update before ./securedrop-admin setup or the latter just finishes without installing dependencies or creating the virtualenv (with no errors)
• with apt cache updated, apt updates start and then the script gets backgrounded somehow? fg gets it back up and running, and it installs and sets up packages FIXME :this could be due to some weird ineraction with Tails apt package persistence hooks, the pre-apt one is triggered just beforehand Update: it is not - disabling the hooks doesn't change the behaviour
• a Python 3.11 virtualenv is set up in the usual spot and the pip requirements install starts, and fails after installing only a few dependencies
• running the pip install command locally, it fails with "[Errno 111] Connection refused" errors. Prefixing with torify allows it to proceed. Weirdly enough we're already doing that in the script, maybe more network instability?Might be a bug or change in behavior for Tails 6 vs 5
• the pip install fails while trying to run setup.py install for cffi, as there is no available C compiler FIXME: there is no 3.9 wheel for 1.14.5, will update to something later
• pip install proceeds and installs requirements successfully, but fails to build and install the securedrop-admin module - long story short, downgraded back to setuptools 56.0.0 with d2to1 :(. It might be worth modernizing the packaging story for securedrop-admin if it's simple but the downgrade gets us to a working securedrop-admin for now.
• securedrop-admin sdconfig works! yaaaay (had to add --force to bypass git version check, as due to network flakiness I had to do a shallow clone of the repo)

[zenmonkeykstop]

OK, securedrop-admin setup now works with minimal changes on Tails 6.0rc1! Some highlights:

• the bootstrap script would suspend after running sudo apt-get update - there wasn't an obvious fix for this and we're running it before the setup command anyway so that has just been dropped.
• The code to check for and eliminate tails 4 venvs now checks for and eliminates tails 5 venvs.
• some python dependencies have been updated, mostly where there were no 3.11 wheels available. We may need to diff review them (and could also update ansible to latest and clear a lot of ignored alerts)
• Random network flakiness is unrelated to the admin tools.

Next up is a full install with a Tails6 workstation using this brach, which will shake out any further issues with Ansible playbooks, the network hook, and the GUI updater.

[zenmonkeykstop]

Looking at this now it's kindof working: in terms of Tails5 vs 6 support there are only 2 prod dependency updates so far (cffi and pyyaml), so it might be feasible to continue to support both by specifying python versions in requirements.txt. The OS detection/venv deletion would need tweaking tho.

[nathandyer]

Started working through a full install this afternoon with @zenmonkeykstop. We were able to work around a few bumps in the road, but I'm now consistently getting stuck on an scp failure when trying to connect to app (strangely, it seems to connect to mon without issue):

TASK [restrict-direct-access : Copy SSH client config file.] *****************************************
task path: /home/amnesia/Persistent/securedrop/install_files/ansible-base/roles/restrict-direct-access/tasks/ssh.yml:2
fatal: [app]: UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via scp: Executing: program /usr/bin/ssh host 10.20.2.2, user (unspecified), command sftp\nOpenSSH_9.2p1 Debian-2+deb12u2, OpenSSL 3.0.11 19 Sep 2023\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 1: Applying options for 10.*\r\ndebug1: /etc/ssh/ssh_config line 4: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 10.20.2.2 is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/amnesia/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/amnesia/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 32932\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Control master terminated unexpectedly\r\nscp: Connection closed\r\n",
    "unreachable": true
}

[zenmonkeykstop]

For my side, I successfully completed a prod VM install using the tails6-ftw branch, but now on subsequent runs the install playbook fails on both servers at Install the grsecurity-patched kernel from the FPF repo:

<x4ukiupj22yk227amlixnuspctmuydevvwareaprjehkpmzdnswfflyd.onion> ESTABLISH SSH CONNECTION FOR USER: vagrant
<x4ukiupj22yk227amlixnuspctmuydevvwareaprjehkpmzdnswfflyd.onion> SSH: EXEC ssh -vvv -o ControlMaster=auto -o ControlPersist=1200 -o ServerAliveInterval=10 -o ServerAliveCountMax=3 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="vagrant"' -o ConnectTimeout=120 -o 'ControlPath="/home/amnesia/.ansible/cp/07403822a4"' x4ukiupj22yk227amlixnuspctmuydevvwareaprjehkpmzdnswfflyd.onion '/bin/sh -c '"'"'echo ~vagrant && sleep 0'"'"''
<h6drfqruyggzv4i7prsnai4vy4zfwu6z4ch3fk42q7jbmnfy43c5tayd.onion> ESTABLISH SSH CONNECTION FOR USER: vagrant
<h6drfqruyggzv4i7prsnai4vy4zfwu6z4ch3fk42q7jbmnfy43c5tayd.onion> SSH: EXEC ssh -vvv -o ControlMaster=auto -o ControlPersist=1200 -o ServerAliveInterval=10 -o ServerAliveCountMax=3 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="vagrant"' -o ConnectTimeout=120 -o 'ControlPath="/home/amnesia/.ansible/cp/644ebc8969"' h6drfqruyggzv4i7prsnai4vy4zfwu6z4ch3fk42q7jbmnfy43c5tayd.onion '/bin/sh -c '"'"'echo ~vagrant && sleep 0'"'"''
<h6drfqruyggzv4i7prsnai4vy4zfwu6z4ch3fk42q7jbmnfy43c5tayd.onion> (0, b'/home/vagrant\n', b"OpenSSH_9.2p1 Debian-2+deb12u2, OpenSSL 3.0.11 19 Sep 2023\r\ndebug1: Reading configuration data /home/amnesia/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 4: Applying options for *\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/amnesia/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/amnesia/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 21823\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n")
<h6drfqruyggzv4i7prsnai4vy4zfwu6z4ch3fk42q7jbmnfy43c5tayd.onion> ESTABLISH SSH CONNECTION FOR USER: vagrant
<h6drfqruyggzv4i7prsnai4vy4zfwu6z4ch3fk42q7jbmnfy43c5tayd.onion> SSH: EXEC ssh -vvv -o ControlMaster=auto -o ControlPersist=1200 -o ServerAliveInterval=10 -o ServerAliveCountMax=3 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="vagrant"' -o ConnectTimeout=120 -o 'ControlPath="/home/amnesia/.ansible/cp/644ebc8969"' h6drfqruyggzv4i7prsnai4vy4zfwu6z4ch3fk42q7jbmnfy43c5tayd.onion '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/vagrant/.ansible/tmp `"&& mkdir "` echo /home/vagrant/.ansible/tmp/ansible-tmp-1707862803.610462-23512-224934042568432 `" && echo ansible-tmp-1707862803.610462-23512-224934042568432="` echo /home/vagrant/.ansible/tmp/ansible-tmp-1707862803.610462-23512-224934042568432 `" ) && sleep 0'"'"''
<x4ukiupj22yk227amlixnuspctmuydevvwareaprjehkpmzdnswfflyd.onion> (0, b'/home/vagrant\n', b"OpenSSH_9.2p1 Debian-2+deb12u2, OpenSSL 3.0.11 19 Sep 2023\r\ndebug1: Reading configuration data /home/amnesia/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 4: Applying options for *\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/amnesia/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/amnesia/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 21827\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n")
<x4ukiupj22yk227amlixnuspctmuydevvwareaprjehkpmzdnswfflyd.onion> ESTABLISH SSH CONNECTION FOR USER: vagrant
<x4ukiupj22yk227amlixnuspctmuydevvwareaprjehkpmzdnswfflyd.onion> SSH: EXEC ssh -vvv -o ControlMaster=auto -o ControlPersist=1200 -o ServerAliveInterval=10 -o ServerAliveCountMax=3 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="vagrant"' -o ConnectTimeout=120 -o 'ControlPath="/home/amnesia/.ansible/cp/07403822a4"' x4ukiupj22yk227amlixnuspctmuydevvwareaprjehkpmzdnswfflyd.onion '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/vagrant/.ansible/tmp `"&& mkdir "` echo /home/vagrant/.ansible/tmp/ansible-tmp-1707862804.2061768-23511-238648797881840 `" && echo ansible-tmp-1707862804.2061768-23511-238648797881840="` echo /home/vagrant/.ansible/tmp/ansible-tmp-1707862804.2061768-23511-238648797881840 `" ) && sleep 0'"'"''
<x4ukiupj22yk227amlixnuspctmuydevvwareaprjehkpmzdnswfflyd.onion> (0, b'ansible-tmp-1707862804.2061768-23511-238648797881840=/home/vagrant/.ansible/tmp/ansible-tmp-1707862804.2061768-23511-238648797881840\n', b"OpenSSH_9.2p1 Debian-2+deb12u2, OpenSSL 3.0.11 19 Sep 2023\r\ndebug1: Reading configuration data /home/amnesia/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 4: Applying options for *\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/amnesia/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/amnesia/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 21827\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n")
Using module file /home/amnesia/Persistent/securedrop/admin/.venv3/lib/python3.11/site-packages/ansible/modules/apt.py
<x4ukiupj22yk227amlixnuspctmuydevvwareaprjehkpmzdnswfflyd.onion> PUT /home/amnesia/.ansible/tmp/ansible-local-23019kjykg9qk/tmpf8nm37xe TO /home/vagrant/.ansible/tmp/ansible-tmp-1707862804.2061768-23511-238648797881840/AnsiballZ_apt.py
<x4ukiupj22yk227amlixnuspctmuydevvwareaprjehkpmzdnswfflyd.onion> SSH: EXEC scp -vvv -o ControlMaster=auto -o ControlPersist=1200 -o ServerAliveInterval=10 -o ServerAliveCountMax=3 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="vagrant"' -o ConnectTimeout=120 -o 'ControlPath="/home/amnesia/.ansible/cp/07403822a4"' /home/amnesia/.ansible/tmp/ansible-local-23019kjykg9qk/tmpf8nm37xe '[x4ukiupj22yk227amlixnuspctmuydevvwareaprjehkpmzdnswfflyd.onion]:/home/vagrant/.ansible/tmp/ansible-tmp-1707862804.2061768-23511-238648797881840/AnsiballZ_apt.py'
<h6drfqruyggzv4i7prsnai4vy4zfwu6z4ch3fk42q7jbmnfy43c5tayd.onion> (0, b'ansible-tmp-1707862803.610462-23512-224934042568432=/home/vagrant/.ansible/tmp/ansible-tmp-1707862803.610462-23512-224934042568432\n', b"OpenSSH_9.2p1 Debian-2+deb12u2, OpenSSL 3.0.11 19 Sep 2023\r\ndebug1: Reading configuration data /home/amnesia/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 4: Applying options for *\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/amnesia/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/amnesia/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 21823\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n")
Using module file /home/amnesia/Persistent/securedrop/admin/.venv3/lib/python3.11/site-packages/ansible/modules/apt.py
<h6drfqruyggzv4i7prsnai4vy4zfwu6z4ch3fk42q7jbmnfy43c5tayd.onion> PUT /home/amnesia/.ansible/tmp/ansible-local-23019kjykg9qk/tmpt7tcqkn7 TO /home/vagrant/.ansible/tmp/ansible-tmp-1707862803.610462-23512-224934042568432/AnsiballZ_apt.py
<h6drfqruyggzv4i7prsnai4vy4zfwu6z4ch3fk42q7jbmnfy43c5tayd.onion> SSH: EXEC scp -vvv -o ControlMaster=auto -o ControlPersist=1200 -o ServerAliveInterval=10 -o ServerAliveCountMax=3 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="vagrant"' -o ConnectTimeout=120 -o 'ControlPath="/home/amnesia/.ansible/cp/644ebc8969"' /home/amnesia/.ansible/tmp/ansible-local-23019kjykg9qk/tmpt7tcqkn7 '[h6drfqruyggzv4i7prsnai4vy4zfwu6z4ch3fk42q7jbmnfy43c5tayd.onion]:/home/vagrant/.ansible/tmp/ansible-tmp-1707862803.610462-23512-224934042568432/AnsiballZ_apt.py'

TASK [grsecurity : Install the grsecurity-patched kernel from the FPF repo.] ***************
task path: /home/amnesia/Persistent/securedrop/install_files/ansible-base/roles/grsecurity/tasks/from_fpf_repo_install_grsec.yml:28
fatal: [app]: UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via scp: Executing: program /usr/bin/ssh host x4ukiupj22yk227amlixnuspctmuydevvwareaprjehkpmzdnswfflyd.onion, user (unspecified), command sftp\nOpenSSH_9.2p1 Debian-2+deb12u2, OpenSSL 3.0.11 19 Sep 2023\r\ndebug1: Reading configuration data /home/amnesia/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 4: Applying options for *\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/amnesia/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/amnesia/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 21827\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Control master terminated unexpectedly\r\nscp: Connection closed\r\n",
    "unreachable": true
}
fatal: [mon]: UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via scp: Executing: program /usr/bin/ssh host h6drfqruyggzv4i7prsnai4vy4zfwu6z4ch3fk42q7jbmnfy43c5tayd.onion, user (unspecified), command sftp\nOpenSSH_9.2p1 Debian-2+deb12u2, OpenSSL 3.0.11 19 Sep 2023\r\ndebug1: Reading configuration data /home/amnesia/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 4: Applying options for *\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/amnesia/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/amnesia/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 21823\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Control master terminated unexpectedly\r\nscp: Connection closed\r\n",
    "unreachable": true
}

NO MORE HOSTS LEFT *************************************************************************

NO MORE HOSTS LEFT *************************************************************************

PLAY RECAP *********************************************************************************
app                        : ok=14   changed=0    unreachable=1    failed=0    skipped=2    rescued=0    ignored=0   
localhost                  : ok=14   changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
mon                        : ok=14   changed=0    unreachable=1    failed=0    skipped=2    rescued=0    ignored=0   

Messing about with Ansible connection configs can sometimes change how and where it consistently fails (for example, with pipelining disabled it always fails earlier for me, just for app, at the SecureBoot check) but it always still fails.

[zenmonkeykstop]

Following up: the failure is consistent (for me) on that one task. Isolating it by running ansible-playbook securedrop-prod.yml --tags "grsec" produces the same result. Disabling controlpersists doesn't seem to make a difference.

[zenmonkeykstop]

This is an OpenSSH problem methinks (Tails 5 has 8, Tails 6 has 9), - it's attempting to use sftp instead of scp for file transfers as per ansible/ansible#78600. The workaround described in said issue clears the problem for me. Alternatively we could update the server sshd_config to enable sftp, but would have to consider ramifications of that first.