--sh-boot --venv PEXes don't respect PEX_PYTHON, PEX_PYTHON_PATH, PEX_PATH if venv exists

[huonw]

Similar to #2725, if a pex is built with --sh-boot --venv, executed at least once with the default configuration (to seed the venv in the PEX_ROOT cache), then certain env vars to control the details of the execution are ignored:

• explicitly selecting a python interpreter with PEX_PYTHON
• constraining the interpreter search with PEX_PYTHON_PATH
• inserting additional requirements/code into the venv via PEX_PATH

This behaviour differs to non---sh-boot where the vars are not ignored. Here's the results of running the reproducer below on my machine:

execution	result with --no-sh-boot	result with --sh-boot
seeding venv interpreter	3.10	3.10
PEX_PYTHON interpreter	3.9	3.10 ❌
PEX_PYTHON_PATH interpreter	3.9	3.10 ❌
PEX_PATH can import cowsay?	✅	❌

Reproducer that tests overriding the PEX_PYTHON* vars to choose Python 3.9 for PEX execution, and also PEX_PATH for inserting and importing a cowsay requirement:

export PEX_ROOT=$(mktemp -d)

pex --version # 2.33.7

pex cowsay -o cowsay.pex

for boot_arg in --no-sh-boot  --sh-boot; do
    echo "# Bootstrap: ${boot_arg}"

    pex "$boot_arg" --venv -o base.pex

    echo "## Seed venv: what default version?"
    ./base.pex -c "import sys; print(sys.version)"

    echo "## Run with Python 3.9 via PEX_PYTHON"
    PEX_PYTHON=$(which python3.9) ./base.pex -c "import sys; print(sys.version)"

    echo "## Run with Python 3.9 via PEX_PYTHON_PATH"
    path_dir=$(mktemp -d)
    ln -s "$(which python3.9)" "$path_dir"

    PEX_EMIT_WARNINGS=false PEX_PYTHON_PATH=$path_dir ./base.pex -c "import sys; print(sys.version)"

    echo "## Run with additional requirements via PEX_PATH"
    PEX_PATH=cowsay.pex ./base.pex -c "import cowsay; print(cowsay.__version__)"

    echo
    echo
done

[jsirois]

It's great you're being thorough here, but I think you could be even more thorough by just noting all PEX env vars not explicitly dealt with in the venv __main__.py will be ignored (I believe the venv script warns for each such case). So that's the conceptual universe to review here beyond the spot checks Pants is effectively doing through its happenstance usage.

[huonw]

Thanks for the tip.

In terms of action, it seems like there's several options:

1. spot-checks: continue playing whack-a-mole and squishing the specific examples, like Avoid fast-path in --sh-boot script when PEX_TOOLS=1 #2726 / Avoid fast-path in --sh-boot script for PEX_PYTHON, PEX_PYTHON_PATH and PEX_PATH #2729
2. deny-list: review the current variables relevant to pex and have a complete deny-list of variables that disable the fast-path if set
3. allow-list: disable the fast path if any pex-y variable is set (i.e. starting with PEX_, _PEX_ or __PEX_), except for an allow-list of those understood by the venv_pex.py / venv __main__.py script and/or known to not make a behaviour difference

I agree that continuing on path 1 isn't very satisfactory. Before I dive down one of 2 or 3, do you have a preference, or alternative suggestion?

I believe the venv script warns for each such case

For this specifically, I note that it doesn't warn for the variables in this particular issue, or some others:

pex/pex/venv/venv_pex.py

Lines 158 to 165 in 57775a4

	"PEX_ROOT",
	"PEX_VENV",
	"PEX_PATH",
	"PEX_PYTHON",
	"PEX_PYTHON_PATH",
	"PEX_VERBOSE",
	"PEX_EMIT_WARNINGS",
	"PEX_MAX_INSTALL_JOBS",