Test for unwanted pyc artifiacts in RPM via CI

[conorsch]

On a whim (OK, was discussing #445 with @eloquence), ran rpmlint against our securedrop-workstation-dom0-config RPM. Here are the results:

[user@disp1758 ~]$ rpmlint QubesIncoming/sd-dev/securedrop-workstation-dom0-config-0.1.2-1.fc25.noarch.rpm 
securedrop-workstation-dom0-config.noarch: W: incoherent-version-in-changelog 0.1.2 ['0.1.2-1.fc25', '0.1.2-1']
securedrop-workstation-dom0-config.noarch: E: dir-or-file-in-opt /opt/securedrop/launcher
securedrop-workstation-dom0-config.noarch: E: dir-or-file-in-opt /opt/securedrop/launcher/sdw-launcher.py
securedrop-workstation-dom0-config.noarch: E: wrong-script-interpreter /opt/securedrop/launcher/sdw-launcher.py /usr/bin/env python3
securedrop-workstation-dom0-config.noarch: E: non-executable-script /opt/securedrop/launcher/sdw-launcher.py 644 /usr/bin/env python3
securedrop-workstation-dom0-config.noarch: E: dir-or-file-in-opt /opt/securedrop/launcher/sdw-launcher.pyc
securedrop-workstation-dom0-config.noarch: E: dir-or-file-in-opt /opt/securedrop/launcher/sdw-launcher.pyo
securedrop-workstation-dom0-config.noarch: E: dir-or-file-in-opt /opt/securedrop/launcher/sdw_updater_gui
securedrop-workstation-dom0-config.noarch: E: dir-or-file-in-opt /opt/securedrop/launcher/sdw_updater_gui/Updater.py
securedrop-workstation-dom0-config.noarch: E: wrong-script-interpreter /opt/securedrop/launcher/sdw_updater_gui/Updater.py /usr/bin/env python3
securedrop-workstation-dom0-config.noarch: E: non-executable-script /opt/securedrop/launcher/sdw_updater_gui/Updater.py 644 /usr/bin/env python3
securedrop-workstation-dom0-config.noarch: E: dir-or-file-in-opt /opt/securedrop/launcher/sdw_updater_gui/Updater.pyc
securedrop-workstation-dom0-config.noarch: E: dir-or-file-in-opt /opt/securedrop/launcher/sdw_updater_gui/Updater.pyo
securedrop-workstation-dom0-config.noarch: E: dir-or-file-in-opt /opt/securedrop/launcher/sdw_updater_gui/UpdaterApp.py
securedrop-workstation-dom0-config.noarch: E: dir-or-file-in-opt /opt/securedrop/launcher/sdw_updater_gui/UpdaterApp.pyc
securedrop-workstation-dom0-config.noarch: E: dir-or-file-in-opt /opt/securedrop/launcher/sdw_updater_gui/UpdaterApp.pyo
securedrop-workstation-dom0-config.noarch: E: dir-or-file-in-opt /opt/securedrop/launcher/sdw_updater_gui/UpdaterAppUi.py
securedrop-workstation-dom0-config.noarch: E: dir-or-file-in-opt /opt/securedrop/launcher/sdw_updater_gui/UpdaterAppUi.pyc
securedrop-workstation-dom0-config.noarch: E: dir-or-file-in-opt /opt/securedrop/launcher/sdw_updater_gui/UpdaterAppUi.pyo
securedrop-workstation-dom0-config.noarch: E: dir-or-file-in-opt /opt/securedrop/launcher/sdw_updater_gui/strings.py
securedrop-workstation-dom0-config.noarch: E: dir-or-file-in-opt /opt/securedrop/launcher/sdw_updater_gui/strings.pyc
securedrop-workstation-dom0-config.noarch: E: dir-or-file-in-opt /opt/securedrop/launcher/sdw_updater_gui/strings.pyo
securedrop-workstation-dom0-config.noarch: E: wrong-script-interpreter /srv/salt/sd/sd-proxy/do-not-open-here /usr/bin/env python3
securedrop-workstation-dom0-config.noarch: E: non-executable-script /srv/salt/sd/sd-proxy/do-not-open-here 644 /usr/bin/env python3
securedrop-workstation-dom0-config.noarch: E: non-executable-script /srv/salt/sd/sys-firewall/sd-copy-rpm-repo-pubkey.sh 644 /bin/bash 
securedrop-workstation-dom0-config.noarch: E: wrong-script-interpreter /srv/salt/securedrop-login /usr/bin/env python3
securedrop-workstation-dom0-config.noarch: E: non-executable-script /srv/salt/securedrop-login 644 /usr/bin/env python3
securedrop-workstation-dom0-config.noarch: E: wrong-script-interpreter /usr/bin/securedrop-admin /usr/bin/env python3
securedrop-workstation-dom0-config.noarch: E: wrong-script-interpreter /usr/share/securedrop-workstation-dom0-config/scripts/configure-environment /usr/bin/env python3
securedrop-workstation-dom0-config.noarch: E: wrong-script-interpreter /usr/share/securedrop-workstation-dom0-config/scripts/destroy-vm /usr/bin/env python3
securedrop-workstation-dom0-config.noarch: E: script-without-shebang /usr/share/securedrop-workstation-dom0-config/scripts/rpmbuilder.Dockerfile
securedrop-workstation-dom0-config.noarch: E: wrong-script-interpreter /usr/share/securedrop-workstation-dom0-config/scripts/securedrop-admin.py /usr/bin/env python3
securedrop-workstation-dom0-config.noarch: E: wrong-script-interpreter /usr/share/securedrop-workstation-dom0-config/scripts/validate_config.py /usr/bin/env python3
securedrop-workstation-dom0-config.noarch: W: no-manual-page-for-binary securedrop-admin
securedrop-workstation-dom0-config.noarch: W: no-manual-page-for-binary securedrop-update
1 packages and 0 specfiles checked; 32 errors, 3 warnings.

Not all of those are must-fix IMO, although the inclusion of .pyc files stands out as a serious issue we should fix. Also willing to update the interpreter to a hardcoded python appropriate to dom0, i.e. /usr/bin/python3 at time of writing.

[eloquence]

although the inclusion of .pyc files stands out as a serious issue we should fix

What's motivating the desire to exclude *.pyc/*.pyo files? The linter does not flag them because of their format (it complains that they're in /opt). As I understand, these are platform-independent bytecode files. If we do not ship them, will there be a permissions issue for scripts run as the GUI user, but from a directory they cannot write to?

Some digging suggests this is part of the Build Root Policy (BRP) brp-python-bytecompile within RPM itself, not anything we do. We can disable this policy by various means (I've found %undefine py_auto_byte_compile works for our build container), but do we want to do this? For now, I've instead explicitly listed the *.pyc and *.pyo files we do implicitly create via the BRP as part of the packaging changes in #445.

Note that Qubes itself also ships the pyc files in its own RPMs, albeit via a __pycache__ directory.

[kushaldas]

The linter does not flag them because of their format (it complains that they're in /opt).

Yes, as rpmlint as a tool follows the Fedora packaging guidelines.

If we do not ship them, will there be a permissions issue for scripts run as the GUI user, but from a directory they cannot write to?

No issues in that.

Some digging suggests this is part of the Build Root Policy (BRP) brp-python-bytecompile within RPM itself, not anything we do. We can disable this policy by various means (I've found %undefine py_auto_byte_compile works for our build container), but do we want to do this? For now, I've instead explicitly listed the *.pyc and *.pyo files we do implicitly create via the BRP as part of the packaging changes in #445.

We can either disable generating those, or delete them before packaging.

Note that Qubes itself also ships the pyc files in its own RPMs, albeit via a pycache directory.

Because Qubes tools are getting installed as proper Python packages/tools. If we follow the standard packaging guide (for python), then we will install in /usr/lib(64)/python3.5/site-packages/ directory, in that case, we can just ship everything together.

[eloquence]

We discussed this briefly at standup today. It sounded like the most compelling reason to avoid shipping *.pyc/*.pyo files is to ensure forward-compatibility of RPMs independent of the system Python version. (Is Python not smart enough to re-generate bytecode as needed?)

I'm happy to disable the BRP as part of #445 with an explanatory comment. CI will fail on that PR if we do nothing, because it more narrowly scopes the list of files that are part of the package, which will cause a "files installed but not packaged" build error due to the implicit generation of *.pyc and *.pyo files.

On the plus side, if that BRP rule fails at any point due to build container updates, CI will just fail again with a very clear error message why.

NB: I won't resolve this issue via #445 due to the other linting issues, which we should also track.

[conorsch]

As mentioned above, #445 addressed the original issue. While it'd be nice to handle the linting requirements, in order to resolve this issue, a CI check on the RPM artifact would be sufficient to guard against regressions:

find -type f -iname '*.rpm' -exec rpm -qpl {} + | grep -iP '\.pyc$'

should show nothing.

[eloquence]

Retitled to clarify remaining scope.

[zenmonkeykstop]

Closing - fixed by work on reproducible images.