Help on writing a "multirun" plugin

[pawamoy]

Following #831, I'm taking a stab at implementing a multirun command for PDM.
Basically, it takes a list of version from the PDM_MULTIRUN_VERSIONS environment variable, and for each:

• it simulates the use command
• then it runs the command given on the command line

import argparse
import os
from pathlib import Path

from pdm.cli import actions
from pdm.cli.hooks import HookManager
from pdm.cli.commands.run import Command as RunCommand, Project

# TODO: just testing, will change
PYTHON_VERSIONS = os.getenv("PDM_MULTIRUN_VERSIONS", "").split()
PYTHON_VERSIONS = PYTHON_VERSIONS or [f"python3.{minor}" for minor in range(7, 12)]


class MultirunCommand(RunCommand):
    """Run a command under multiple Python versions."""

    def handle(self, project: Project, options: argparse.Namespace) -> None:
        for python_version in PYTHON_VERSIONS:
            actions.do_use(
                project,
                python=python_version,
                first=True,
                ignore_remembered=False,
                hooks=HookManager(project, skip=options.skip),
            )
            print(Path(".pdm.toml").read_text())  # debug
            try:
                super().handle(project, options)
            except SystemExit as exit:
                if exit.code:
                    raise


def multirun(core):
    core.register_command(MultirunCommand, "multirun")

However it seems that PDM reads .pdm.toml before running pdm.cli.commands.run.Command.handle (super().handle in my snippet), so the Python version is always the same:

% python -m pdm multirun python -V
Using the last selection, add '-i' to ignore it.
Using Python interpreter: /home/pawamoy/.basher-packages/pyenv/pyenv/shims/python3.7 (3.7)
Updating executable scripts...
[python]
path = "/home/pawamoy/.basher-packages/pyenv/pyenv/shims/python3.7"

Python 3.7.15
Using the last selection, add '-i' to ignore it.
Using Python interpreter: /home/pawamoy/.basher-packages/pyenv/pyenv/shims/python3.8 (3.8)
Updating executable scripts...
[python]
path = "/home/pawamoy/.basher-packages/pyenv/pyenv/shims/python3.8"

Python 3.7.15
Using the last selection, add '-i' to ignore it.
Using Python interpreter: /home/pawamoy/.basher-packages/pyenv/pyenv/shims/python3.9 (3.9)
Updating executable scripts...
[python]
path = "/home/pawamoy/.basher-packages/pyenv/pyenv/shims/python3.9"

Python 3.7.15
Using the last selection, add '-i' to ignore it.
Using Python interpreter: /home/pawamoy/.basher-packages/pyenv/pyenv/shims/python3.10 (3.10)
Updating executable scripts...
[python]
path = "/home/pawamoy/.basher-packages/pyenv/pyenv/shims/python3.10"

Python 3.7.15
Using the last selection, add '-i' to ignore it.
Using Python interpreter: /home/pawamoy/.basher-packages/pyenv/pyenv/shims/python3.11 (3.11)
Updating executable scripts...
[python]
path = "/home/pawamoy/.basher-packages/pyenv/pyenv/shims/python3.11"

Python 3.7.15

Is there a way to force-reload the configuration from .pdm.toml?
Or is there another attribute of project I should modify so the Python version change is taken into account?

Also, I would like to hide the output of actions.do_use, is this possible?

[pawamoy]

I'm able to discard the output with this:

old_echo = project.core.ui.echo
project.core.ui.echo = lambda *args, **kwargs: None

Though still no luck with getting the right Python version to run 😕

[frostming]

The project.environment is set and cached, you need to clear that value after switching:

project.environment = None

And it will work fine.

[pawamoy]

Perfect, thanks a lot!

[pawamoy]

OK there's still an issue: there's some kind of leakage between the different Python processes. A multirun ends up with cross-versions errors such as:

Traceback (most recent call last):
  File "/media/data/dev/pdm-multirun/__pypackages__/3.11/lib/duty/validation.py", line 10, in <module>
    from functools import cached_property  # type: ignore
ImportError: cannot import name 'cached_property' from 'functools' (/home/pawamoy/.basher-packages/pyenv/pyenv/versions/3.7.15/lib/python3.7/functools.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/media/data/dev/pdm-multirun/__pypackages__/3.11/bin/duty", line 5, in <module>
    from duty.cli import main
  File "/media/data/dev/pdm-multirun/__pypackages__/3.11/lib/duty/cli.py", line 24, in <module>
    from duty.validation import validate
  File "/media/data/dev/pdm-multirun/__pypackages__/3.11/lib/duty/validation.py", line 12, in <module>
    from cached_property import cached_property  # type: ignore  # noqa: WPS440
ModuleNotFoundError: No module named 'cached_property'

Note how Python 3.11 is used but the Python 3.7 version of functools is accessed (resulting in import errors).

I believe commands are ran in subprocesses so I don't understand how this can happen.

[pawamoy]

Ah, found it: I was running project.environment = None too late, PDM and Python were not in sync. I now run it before running actions.do_use() and it works.

[noirbizarre]

That's awesome !
I started something similar but it seems better to contribute to this one 👌🏼

[pawamoy]

Glad you like it! Contributions really welcome 😄!
Repository is here by the way: https://github.com/pawamoy/pdm-multirun (for reference)