Skip to content

patch: v0.4.1 architecture fix #1035

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 2, 2025

Conversation

asoplata
Copy link
Collaborator

@asoplata asoplata commented Apr 1, 2025

This second emergency fix today fixes a very sneaky bug introduced in the packaged but not source versions of HNN-core for MacOS and Linux installs on x86_64 processors (not arm64 or Windows). In the name of sanity, I am going to switch the version to v0.4.1 thereby introducing semantic versioning, in order to provide a newer but still close version to Pypi.

Previously, there was a small inconsistency unnoticed by me, and introduced by our support for arm64. The existing release instructions https://github.com/jonescompneurolab/hnn-core/wiki/How-to-make-a-release tell you that, before uploading the distributable package, you should

rm -rf hnn-core/hnn_core/mod/x86_64

Since similar functionality is in our root Makefile, but the latter includes code for handling our new hnn_core/mod/arm64 directory, I used make clean instead of the above command. However, our Makefile's version is very slightly different:

clean :
	rm -rf hnn_core/mod/x86_64/*
	rm -rf hnn_core/mod/arm64/*

Spot the difference? Our makefile uses dir/* instead of dir above. This is important, and I learned something today: rm -rf dir/* does NOT delete "hidden" files and directories inside a directory, see

https://unix.stackexchange.com/questions/289391/why-does-my-command-rm-rf-not-delete-all-the-files-in-a-directory

This is an issue for us because, as it turns out, when we compile our NEURON objects into mod/<arch>, it generates a hidden directory located at mod/<arch>/.libs. For example, if you install hnn_core through a local source version, this create a hidden (meaning beginning with a dot) directory at either hnn_core/mod/arm64/.libs or hnn_core/mod/x86_64/.libs.

This hnn_core/mod/<arch>/.libs directory contains a generated library called libnrnmech.so. Thus, because I was using make clean, the distributable I uploaded to Pypi includes a
hnn_core/mod/arm64/.libs/libnrnmech.so library object. In the case of Windows and MacOS on arm64, this did not cause any problems. However, on x86_64 MacOS and Linux, when a user tries to run a simulation for the first time, it appears that this library is attemptively used to build the compiled NEURON mechanisms. However, since it's the wrong architecture, it fails, and the system is unable to build the NEURON mechanisms, and therefore unable to simulate.

This is not an issue in our tests because our tests are always installing from source from scratch, and its hnn_core/mod folder never contains architecture-specific artifacts.

Again, I apologize for the misstep. For my upcoming updates to how to make a release, I am definitely adding "personally test any distributables across all used architectures, even if the Github Actions unit tests pass across architectures" as a step. Relatedly, this bug is a very good reason to soon invest in pushing new releases directly from Github release builds themselves, instead of personally-built distributable files.

This second emergency fix today fixes a very sneaky bug introduced in
the *packaged* but not source versions of HNN-core for MacOS and Linux
installs on x86_64 processors (not arm64 or Windows). In the name of
sanity, I am going to switch the version to v0.4.1 thereby introducing
semantic versioning, in order to provide a newer but still close version
to Pypi.

Previously, there was a small inconsistency unnoticed by me, and
introduced by our support for arm64. The existing release instructions
https://github.com/jonescompneurolab/hnn-core/wiki/How-to-make-a-release
tell you that, before uploading the distributable package, you should
```
rm -rf hnn-core/hnn_core/mod/x86_64
```
Since similar functionality is in our root Makefile, but the latter
includes code for handling our new `hnn_core/mod/arm64` directory, I used
`make clean` instead of the above command. However, our Makefile's
version is very slightly different:
```
clean :
	rm -rf hnn_core/mod/x86_64/*
	rm -rf hnn_core/mod/arm64/*
```
Spot the difference? Our makefile uses `dir/*` instead of `dir`
above. This is important, and I learned something today: `rm -rf dir/*`
does NOT delete "hidden" files and directories inside a directory, see

https://unix.stackexchange.com/questions/289391/why-does-my-command-rm-rf-not-delete-all-the-files-in-a-directory

This is an issue for us because, as it turns out, when we compile our
NEURON objects into `mod/<arch>`, it generates a hidden directory
located at `mod/<arch>/.libs`. For example, if you install hnn_core
through a local source version, this create a hidden (meaning beginning
with a dot) directory at either `hnn_core/mod/arm64/.libs` or
`hnn_core/mod/x86_64/.libs`.

This `hnn_core/mod/<arch>/.libs` directory contains a generated library
called `libnrnmech.so`. Thus, because I was using `make clean`, the
distributable I uploaded to Pypi includes a
`hnn_core/mod/arm64/.libs/libnrnmech.so` library object. In the case of
Windows and MacOS on arm64, this did not cause any problems. However, on
x86_64 MacOS and Linux, when a user tries to run a simulation for the
first time, it appears that this library is attemptively used to build
the compiled NEURON mechanisms. However, since it's the wrong
architecture, it fails, and the system is unable to build the NEURON
mechanisms, and therefore unable to simulate.

This is not an issue in our tests because our tests are always
installing from source from scratch, and its `hnn_core/mod` folder never
contains architecture-specific artifacts.

Again, I apologize for the misstep. For my upcoming updates to how to
make a release, I am definitely adding "personally test any
distributables across all used architectures, even if the Github Actions
unit tests pass across architectures" as a step. Relatedly, this bug is
a very good reason to soon invest in pushing new releases directly from
Github release builds themselves, instead of personally-built
distributable files.
@asoplata asoplata force-pushed the patch-v0.4.1-arch branch from dfdbcb8 to 75cd5f8 Compare April 2, 2025 00:00
@asoplata asoplata merged commit a2f98d3 into jonescompneurolab:master Apr 2, 2025
12 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant