Remnux upgrade v2023.35.1 > v2023.36.1 Fails

[ShanHolo]

Hello world

When I tried to upgrade to 2023.36.1 I had the following issues

Update returned exit code not zero
Error: Update returned exit code not zero
at ChildProcess. (/snapshot/remnux-cli/remnux-cli.js:571:23)
at ChildProcess.emit (events.js:315:20)
at maybeClose (internal/child_process.js:1021:16)
at Process.ChildProcess._handle.onexit (internal/child_process.js:286:5)

Checking the saltstack log I figured out the following:

file_|-remnux-addon-version-file_|-/etc/remnux-version_|-managed:
id: remnux-addon-version-file
run_num: 780
sls: remnux.addon
changes: {}
comment: 'One or more requisite failed: remnux.python3-packages.remnux-python3-packages'
duration: 0.005
name: /etc/remnux-version
result: false

\ packages: pyelftools\n Attempting uninstall: pyelftools\n Found existing
\ installation: pyelftools 0.29 WARNING: Error parsing requirements for pyelftools:
\ [Errno 2] No such file or directory: '/usr/local/lib/python3.8/dist-packages/pyelftools-0.29.dist-info/METADATA'\n
\ WARNING: No metadata found in /usr/local/lib/python3.8/dist-packages\n
ERROR: Cannot uninstall pyelftools 0.29, RECORD file not found. You might be
\ able to recover from this via: 'pip install --force-reinstall --no-deps pyelftools==0.29'."
duration: 2328.396
name: pyelftools
result: false

test_|-remnux-dedicated_|-remnux-dedicated_|-nop:
id: remnux-dedicated
run_num: 837
sls: remnux.dedicated
changes: {}
comment: 'One or more requisite failed: remnux.addon.remnux-addon-version-file'
duration: 0.004
name: remnux-dedicated
result: false

test_|-remnux-python3-packages_|-remnux-python3-packages_|-nop:
id: remnux-python3-packages
run_num: 413
sls: remnux.python3-packages
changes: {}
comment: 'One or more requisite failed: remnux.python3-packages.pyelftools.remnux-python3-packages-pyelftools'
duration: 0.033
name: remnux-python3-packages
result: false

Just in case I attach the saltstack log:

saltstack.log

[digitalsleuth]

Hi @ShanHolo , a new version was released just after you posted this issue which should resolve the problem you encountered. Can you try your install again?

[ShanHolo]

Hi @digitalsleuth thanks for your quick response. I tried the upgrade again with the following errors:

pdate returned exit code not zero
Error: Update returned exit code not zero
at ChildProcess. (/snapshot/remnux-cli/remnux-cli.js:571:23)
at ChildProcess.emit (events.js:315:20)
at maybeClose (internal/child_process.js:1021:16)
at Process.ChildProcess._handle.onexit (internal/child_process.js:286:5)

Errors on /var/cache/remnux/cli/v2023.37.1/saltstack.log

file_|-remnux-addon-version-file_|-/etc/remnux-version_|-managed:
id: remnux-addon-version-file
run_num: 784
sls: remnux.addon
changes: {}
comment: 'One or more requisite failed: remnux.python3-packages.remnux-python3-packages'
duration: 0.004
name: /etc/remnux-version
result: false

\ packages: pyelftools\n Attempting uninstall: pyelftools\n Found existing
\ installation: pyelftools 0.29 WARNING: Error parsing requirements for pyelftools:
\ [Errno 2] No such file or directory: '/usr/local/lib/python3.8/dist-packages/pyelftools-0.29.dist-info/METADATA'\n
WARNING: No metadata found in /usr/local/lib/python3.8/dist-packages\n
ERROR: Cannot uninstall pyelftools 0.29, RECORD file not found. You might be
\ able to recover from this via: 'pip install --force-reinstall --no-deps pyelftools==0.29'."
duration: 2620.682
name: pyelftools
result: false

test_|-remnux-dedicated_|-remnux-dedicated_|-nop:
id: remnux-dedicated
run_num: 841
sls: remnux.dedicated
changes: {}
comment: 'One or more requisite failed: remnux.addon.remnux-addon-version-file'
duration: 0.003
name: remnux-dedicated
result: false

test_|-remnux-python3-packages_|-remnux-python3-packages_|-nop:
id: remnux-python3-packages
run_num: 417
sls: remnux.python3-packages
changes: {}
comment: 'One or more requisite failed: remnux.python3-packages.pyelftools.remnux-python3-packages-pyelftools'
duration: 0.003
name: remnux-python3-packages
result: false

[geeksailor]

Hi @ShanHolo!
I'll link my ticket but it has to do with pyelftools 0,29 folder existing in the dist-packages with no content, confusing salt. Try
sudo rm -rf /usr/local/lib/python3.8/dist-packages/pyelftools-029.dist-info to clear it out. Without it seeing a folder there it should work to try and run an update again.

#148

@digitalsleuth Is there a way in salt to test if the 0.29 folder exists already and if it does for salt to remove it prior to trying to update pyelftool? That might fix this issue for some.

[digitalsleuth]

Hi @ShanHolo , the release you attempted is missing one command, which is being pushed to the next release. It should be out shortly.
@geeksailor your ticket didn't have to do with pyelftools specifically, just that the pip/wheel/setuptools issue got hung up on that package vice others. That one should also be resolved with the upcoming release.

[ShanHolo]

Thanks a lot for your appreciated support

Trying to update again to the version 37.2 I had the similar issue. @digitalsleuth did you mean the release 37.3? right

Thanks again.

[digitalsleuth]

@ShanHolo , yes 37.3 is the upcoming release, which hopefully we can get out today.

[geeksailor]

@digitalsleuth @ShanHolo Did the 37.3 do anything with Thug because I got a bunch of errors today relating to Thug and Speakeasy while trying to run the update. Otherwise I guess I might have done something to screw it up and the update process.

[digitalsleuth]

@geeksailor Can you provide your log file so I can see the errors? I know Thug released an update yesterday, but speakeasy hasn't had anything recently updated.

[geeksailor]

@digitalsleuth here's the grepped version. I can grab you the full stack too if you wish
saltstack_grepped_091523.txt

[geeksailor]

Ran sudo pip3 show thug and got this spaghetti mess.
pip3_show_thug.txt
So I ran a sudo pip3 show setuptools since that seems to be the culprit and I got this
pip3_show_setuptools.txt

and a pip3 list for both of them for good measure

sudo pip3 list | grep setuptools
WARNING: No metadata found in /usr/local/lib/python3.8/dist-packages
WARNING: No metadata found in /usr/local/lib/python3.8/dist-packages
setuptools 68.1.2
setuptools-rust 1.7.0
remnux@ubuntu:/var/cache/remnux/cli/v2023.37.1$ sudo pip3 list | grep thug
WARNING: No metadata found in /usr/local/lib/python3.8/dist-packages
WARNING: No metadata found in /usr/local/lib/python3.8/dist-packages
thug 5.5

The Warning no metadata warning seems like the setuptools folder is broken and it is breaking everything else. Gonna go check it out
EDIT: Turns out I have 4 versions of Setuptools in my python3.8 dist-packages folder, 68.12, 68.2.0,68.2.1,and 68.2.2, an 68.1.2 only has the REQUESTED file in it and nothing else. Seems like it got removed for the most part minus that file and the folder and might be screwing up PIP when it does an inventory. Gonna try removing it and see what happens.

EDIT 2: Ok that got rid of the spaghetti of errors for doing the show on thug and setuptools. Gonna try running an update again and see if it works. Not sure what happened there to leave those scraps behind but hopefully it fixes it.

[geeksailor]

Side note, I am up to Remnux v2023.37.1 but I still see folders for 35.1 and 36.1 in my /var/cache/remnux/cli/v2023-3* directory. Should those have been auto cleaned up or is that something manual that I can remove?

[digitalsleuth]

@geeksailor Given that you've made a number of changes to your python setup (ie. manually deleting folders under /usr/local/lib/python3.8), this may be an issue with some of the changes you made. However, I will need the full saltstack.log file so I can see what occurred around the installation.

As for the over directories under /var/cache, you can delete 35.1 and 36.1, they don't get cleaned up because they retain the log files.

[geeksailor]

Ran another update and still have problems I think similar to the first. Haven't done a comparison yet.
Grepped version:
saltstack_grepped_091523_1311.txt
Full log:
saltstack.log

[digitalsleuth]

@geeksailor Can you try the following, exactly as below:
sudo python3 -m pip install 'setuptools=68.2.2' --force-reinstall

Once that is done, please take a look in the /usr/local/lib/python3.8/dist-packages/setuptools-68.2.2.dist-info/ folder and list off the items in there?

[geeksailor]

@digitalsleuth Will run it now. Be back in a minute with the results