Propagating the target compiler to subcommands

[straight-shoota]

Since #14953 the compiler can run external executables as subcommands as an alternative to builtin commands.

This kind of subcommand delegation needs to ensure consistency in the compiler version being used.
For example, let's run path/to/crystal subcommand which calls crystal-subcommand from somewhere in $PATH. The subcommand needs to be aware that the target compiler is path/to/crystal, not whatever crystal is in $PATH.

The current solution from #14953 passes the environment variable $CRYSTAL to the subcommand which has the full path to the compiler.

I don't think it works very well. It requires the subcommand to recognize the environment variable. But it must also expect it to be unset when the command is executed directly (as crystal-subcommand). And this trickles down to other processes that the subcommand might spawn. They typically should also be aware of the target crystal compiler. This is practically impossible to realize with the $CRYSTAL environment variable.

git uses a similar mechanism to run subcommands and handles propagating the target instance in a different way: It prepends the path to the parent directory of the executable to $PATH. That means if the subcommand process calls git, it'll resolve to the original one. Path lookup just works when the subcommand is executed directly.

Additionally, git also puts this directory path in the environment variable $GIT_EXEC_PATH. I'm not familiar with the specific details when one is used over the other, but it looks like the prepended $PATH and $GIT_EXEC_PATH are identical.

I think we should add the compiler's parent directory to $PATH. This feature is still brand new and AFAIK not actively used, so I think it should be acceptable to replace the $CRYSTAL variable. We could consider keeping it along, but if we do that, we'll need to do that for a long time. I think it's better to drop it in order to avoid new subcommands ever depend on it.

Introducing a separate $CRYSTAL_EXEC_PATH could also be valuable. I believe it could be a more generic replacement to $CRYSTAL_SPEC_COMPILER_BIN which we use in the spec suite.

[straight-shoota]

I think we should also drop the current mechanism with the CRYSTAL environment variable.

That'll break existing command integrations relying on this veriable. Considering that the entire subcommand feature is new in 1.14 I don't expect much use of it though.

The trouble is that there's really no mechanism to deprecate an environment variable. If we keep supporting it, it'll inevitably be used. And then we effectively can never drop it, or have a breaking change later on when there's potentially more integrations depending on it.
Breaking it this early is acceptable and the best cause of action in my opinion.

[straight-shoota]

I figure we also need a variable $CRYSTAL_EXEC_PATH to keep a reference to the original compiler path, which may not be identical to the process' executable path. For example the latter would be .build/crystal but the command that the user runs is actually bin/crystal. And I suppose we want to keep using bin/crystal instead of the executable path directly.

[bcardiff]

An upside of the $CRYSTAL env variable is that if the compiler is not crystal but something else crystal-1.0, crystal-1.14, crystal-2 then the external commands still works.

I don't mind prepending to the $PATH and having a $CRYSTAL_EXEC_PATH, but I don't see how we enable binaries named differently.

$CRYSTAL is also used in shards as a convention, so I thought it was convenient to keep the same pattern. (which happens to be the same as in cabal with $CABAL)

If no one thinks that supporting different names of the compiler binary is needed then 👍

[straight-shoota]

That's an interesting point 🤔
I'm not sure if there are actually good use cases, though. Usually the compiler entrypoint is called crystal. It's either bin/crystal or the build artifact from make crystal. Technically the artifact name is configurable with CRYSTAL_BIN=crystal-1.0, or the file can be renamed afterwards. Does anybody do that though?

I've sure used names with qualifiers like crystal-1.14 to distinguish different variants of the compiler. But they've always been symlinks to either bin/crystal or the binary from an install location. So the actual entrypoint is still called crystal.

With regards to targeting subcommands, this also raises the question whether crystal-1.14 ext should delegate to crystal-1.14-ext or crystal-ext or both? 🤔

My gut feeling is that this qualifier thing might be more complexity than needed for practical use cases. But I'm really not sure about it.

[bcardiff]

With regards to targeting subcommands, this also raises the question whether crystal-1.14 ext should delegate to crystal-1.14-ext or crystal-ext or both?

I would say the later. crystal-ext could target multiple crystal versions at the same time.

--

I'm not sure if restricting the packaging to have symlinks to a crystal binary is good.

I don't know if any packaging does that.

Using the same convention in shards seems reasonable to me. It could be that $CRYSTAL has the full path or just the name of we are updating the $PATH. Yet others tweaking the $PATH could lead to mismatch binaries.

I like that using just crystal will work in most cases by tweaking the $PATH. But I would not like to loose the ability to have a solution that always work.

[straight-shoota]

With regards to targeting subcommands, this also raises the question whether crystal-1.14 ext should delegate to crystal-1.14-ext or crystal-ext or both?

I would say the later. crystal-ext could target multiple crystal versions at the same time.

Yeah, for custom local extensions in your path, that seems right.
But for core parts of the compiler that are implemented as subcommands, it would make a lot of sense to build them with the same naming scheme as the compiler itself: If the compiler itself is named crystal-1.14, the playground would be called as crystal-1.14 play and delegate to the subcommand crystal-1.14-play.
In this context it seems odd if the subcommand would be crystal-play because if it's in path, it would lead to the same naming conflict as crystal would if multiple versions are installed. And I believe that's the only reason why you would want to call the compiler anything else than crystal.

[bcardiff]

Ok. Then both. Current executable name and then crystal-.

For internal commands we could rely on wrapper scripts setting the path thou. But is not different from relying on symlinks I guess.

So... both?

[RX14]

Priority to executable-name-ext then running crystal-ext makes sense to me.

[ysbaddaden]

Coming back to this:

If we start splitting the single crystal binary into subcommands, having a $CRYSTAL_EXEC_PATH and have it point to a directory (and extend $PATH to search into this path by default) shall prove simpler than a $CRYSTAL that points to a single executable.

In case of multiple versions installed, it's easier to have the subcommands in different directories for each version. This way both internal and external commands can expect $CRYSTAL_EXEC_PATH/crystal-tool-format to exist (no magic, no guesses, no search).

We can still have global links or shell shims to each crystal executable (e.g. ~/bin/crystal-1.2 -> /usr/src/crystal-1.2.0/bin/crystal).

[zw963]

Who can explain what are the really benefits of breaking down into lots of sub commands?

Crystal is not git, latter write use C, and glued together use BASH or others, why must Crystal need do this?

Why we introduce such a complicated thing? support use ext to run external tools is okay, why split format into sub command, and why handle version things?

[ysbaddaden]

For example:

We currently need to compiler EVERYTHING: the compiler (ok) + the doc generator (huh?) + each individual tool: the playground (huh?), the formatter (huh?), ...

With distinct executables (at least for development) we could build the compiler OR the doc generator OR the formatter OR ... It would reduce the number of things to compile, improve the compilation times thus the developer feedback loop.

Bonus: it creates clear boundaries between each tool.

[bcardiff]

In case of multiple versions installed, it's easier to have the subcommands in different directories for each version. This way both internal and external commands can expect $CRYSTAL_EXEC_PATH/crystal-tool-format to exist (no magic, no guesses, no search).

Why would we need the path of a specific tool? I would do $CRYSTAL [external-command] and leave the resolution to the crystal binary.

I think is a little bit more limited to use $CRYSTAL_EXEC_PATH as opposed to $CRYSTAL. But we can go either if you are still not aligned with $CRYSTAL.