Can't cd in test-command on Windows

[gpongelli]

Description

Hi,
looking at the docs, seems that's possible to have multiline commands working on all OSes, writing things in format like cmd1 && cmd2 or

# Multi-line example - join with && on all platforms
CIBW_TEST_COMMAND: >
  pytest {package}/tests &&
  python {package}/test.py

In a GitHub action I've the following test instruction:

          CIBW_TEST_COMMAND : |
            cd {project} && 
            poetry install --with devel --sync --no-root && 
            poetry run tox --skip-pkg-install -e py${{ matrix.python-version }}-${{ matrix.tox_os }}

it works correctly for MacOS and for Linux, so each row is executed correctly and I can see the log also of the third step (tox launches pytest under the hood), but under Windows OS I get this error

+ cd D:\a\pycode128\pycode128 && 
  poetry install --with devel --sync --no-root && 
  poetry run tox --skip-pkg-install -e py3.11-win
  
  The syntax of the command is incorrect.

Running poetry run tox --skip-pkg-install -e py3.11-win locally on my machine it works as expected.

I've also tried without &&

          CIBW_TEST_COMMAND : |
            cd {project}
            poetry install --with devel --sync --no-root
            poetry run tox --skip-pkg-install -e py${{ matrix.python-version }}-${{ matrix.tox_os }}

but seems nothing happens and, most dangerous, it result as passed 😮

+ cd D:\a\pycode128\pycode128
  poetry install --with devel --sync --no-root
  poetry run tox --skip-pkg-install -e py3.10-win
  
                                                             ✓ 30.36s

✓ cp310-win_amd64 finished in 80.05s

Some links:
Execution without && : https://github.com/gpongelli/pycode128/actions/runs/4565065623/jobs/8055715603
Github workflow: https://github.com/gpongelli/pycode128/blob/2.5.0/.github/workflows/complete.yml

Execution with &&: https://github.com/gpongelli/pycode128/actions/runs/4566789867/jobs/8059823600#step:27:436
Github Workflow: https://github.com/gpongelli/pycode128/blob/test_cibw_failing/.github/workflows/complete.yml

ps:
the test execution must all fail on all OSes due to defect into code, spot by pytest

Thanks.

Build log

https://github.com/gpongelli/pycode128/actions/runs/4566789867/jobs/8059823600#step:27:436

CI config

https://github.com/gpongelli/pycode128/blob/test_cibw_failing/.github/workflows/complete.yml

[henryiii]

The problem isn't the &&, it's the newlines. This should work:

          CIBW_TEST_COMMAND : >
            cd {project} && 
            poetry install --with devel --sync --no-root && 
            poetry run tox --skip-pkg-install -e py${{ matrix.python-version }}-${{ matrix.tox_os }}

(Notice you changed > to |!)

It would be even better in your pyproject.toml, but the expansion would be missing. Do you really want to make a poetry environment that you install tox into to make a tox environment to tests something already in an environment in cibuildwheel, though?

[gpongelli]

Do you really want to make a poetry environment that you install tox into to make a tox environment to tests something already in an environment in cibuildwheel, though?

yes, because in this way I can:

• manage dependencies' version with poetry , as a single "point of truth" instead of having some package's specified into requirements.txt and other into setup.cfg
• build package with poetry
• run tox environments to format, lint, test, release calling tools from within poetry env
• call all the tools also outside tox
• be able to run same commands, with or without tox, on all OSes and on CI execution. this also applies to cibuildwheel commands.

I know it's redundant, but it's the only way I found to have all the above points checked.

I've then changed with > character, now the error now is

+ cd D:\a\pycode128\pycode128 &&  poetry install --with devel --sync --no-root &&  poetry run tox --skip-pkg-install -e py3.10-win

  Poetry could not find a pyproject.toml file in C:\Users\runneradmin\AppData\Local\Temp\cibw-run-aec8iccu\cp310-win_amd64\test_cwd or its parents

Error: Command cd D:\a\pycode128\pycode128 &&  poetry install --with devel --sync --no-root &&  poetry run tox --skip-pkg-install -e py3.10-win

seems that the change directory command, before poetry command, is not executed.
Is there a way to solve this ?

Thanks.

[henryiii]

Isn't "dir" the way to change directories in Windows? We also strongly recommend running the tests from a different directory so you don't pick up build artifacts. Maybe point at or copy the files you need, pyproject.toml and possibly tox.ini (if you aren't placing it inside pyproject.toml)?

[gpongelli]

dir shows the folder's content, as ls does under linux.

it's cd even under windows.

[henryiii]

Ahh, oops, I remembered something was different, but yeah, it's ls -> dir. Hmm, have you tried it locally? Does cd subdir && dir print the subdir? (my Windows machine is in pieces currently)

[gpongelli]

from within a powershell, the && led to error: The token '&&' is not a valid statement separator in this version.

from within a git bash, cd subdir && dir prints subdir's content and then the prompt is into subdir.

[gpongelli]

as found here, in powershell 5 commands can be chained with ;, with powershell 7 && can be used.
I've powershell 5 and, chaining with ; , the command must exist on one line.

instead, for cmd, the && works but again all on one line.

which is the process run by cibuildwheels ? powershell? cmd ?

[henryiii]

It should be cmd. And it is on one line, > merges lines in YAML. Before, with |, it was on multiple lines.

[gpongelli]

tested all the cases, no one works 😞

using ; on one line

commit gpongelli/pycode128@fdfc543
run https://github.com/gpongelli/pycode128/actions/runs/4572200081/jobs/8071238497
error

+ cd D:\a\pycode128\pycode128 ; poetry install --with devel --sync --no-root ; poetry run tox --skip-pkg-install -e py3.9-win
  The system cannot find the path specified.

Using && on one line

commit gpongelli/pycode128@c2aa9d0
run https://github.com/gpongelli/pycode128/actions/runs/4572051909/jobs/8070943467
error:

+ cd D:\a\pycode128\pycode128 && poetry install --with devel --sync --no-root && poetry run tox --skip-pkg-install -e py3.9-win

  Poetry could not find a pyproject.toml file in C:\Users\runneradmin\AppData\Local\Temp\cibw-run-5_149aro\cp39-win_amd64\test_cwd or its parents

using && on multiple line

commit gpongelli/pycode128@165a88c
run https://github.com/gpongelli/pycode128/actions/runs/4566789867/jobs/8059823600
error

+ cd D:\a\pycode128\pycode128 && 
  poetry install --with devel --sync --no-root && 
  poetry run tox --skip-pkg-install -e py3.9-win

  The syntax of the command is incorrect.

any idea on how to solve? is this case tested into cibuildwheel's test?

thanks.

[joerick]

This is weird... I'd expect Using && on one line above to work. We do test command chaining in cibuildwheel's test suite, but on the before-build command, and not with a 'cd'.

https://github.com/pypa/cibuildwheel/blob/cfc636f50fac9395ace92740f04dc573b03d27ec/test/test_before_build.py#L47-L49

Is there something weird happening between cd and &&? perhaps you could try cd {project} && dir && poetry ... and see where dir thinks the cwd is.

[gpongelli]

Here it is, commit gpongelli/pycode128@cb15d66

execution https://github.com/gpongelli/pycode128/actions/runs/4579006619/jobs/8086321785

Error

+ cd D:\a\pycode128\pycode128 && dir && poetry install --with devel --sync --no-root && poetry run tox --skip-pkg-install -e py3.11-win

   Volume in drive C is Windows

   Volume Serial Number is EEDD-FD89

   Directory of C:\Users\runneradmin\AppData\Local\Temp\cibw-run-wmjd8g8k\cp311-win_amd64\test_cwd

  03/31/2023  08:46 PM    <DIR>          .
  03/31/2023  08:46 PM    <DIR>          ..
  03/31/2023  08:46 PM               767 test_fail.py

                 1 File(s)            767 bytes
                 2 Dir(s)  83,148,328,960 bytes free

  Poetry could not find a pyproject.toml file in C:\Users\runneradmin\AppData\Local\Temp\cibw-run-wmjd8g8k\cp311-win_amd64\test_cwd or its parents

  Error: Command cd D:\a\pycode128\pycode128 && dir && poetry install --with devel --sync --no-root && poetry run tox --skip-pkg-install -e py3.11-win failed with code 1. None

seems cd isn’t working, am I right?
Is there a workaround or should I wait for a fix on cibuildwheel?

[joerick]

Yeah, it looks like cd is the issue.

Looking on stack overflow, you might need the /d flag?

[joerick]

The difference being that cd on windows won't change drive without /d (and apparently, silently fail as well!)

[gpongelli]

Yeah, it looks like cd is the issue.

Looking on stack overflow, you might need the /d flag?

You got it 🚀
I didn’t know about /d .

The difference being that cd on windows won't change drive without /d (and apparently, silently fail as well!)

That’s very weird, but it’s windows like 😅

Execution https://github.com/gpongelli/pycode128/actions/runs/4582036464/jobs/8091990316

cd /d D:\a\pycode128\pycode128 && dir && poetry install --with devel --sync --no-root && poetry run tox --skip-pkg-install -e py3.11-win

 Volume in drive D is Temporary Storage

 Volume Serial Number is 0ED2-4924

 Directory of D:\a\pycode128\pycode128
04/01/2023  07:44 AM    <DIR>          .
04/01/2023  07:41 AM    <DIR>          ..
04/01/2023  07:41 AM               436 .editorconfig
04/01/2023  07:41 AM    <DIR>          .github
04/01/2023  07:41 AM             1,513 .gitignore
04/01/2023  07:41 AM               216 .gitmodules
04/01/2023  07:41 AM             1,075 .pre-commit-config.yaml
04/01/2023  07:41 AM             1,294 .yamllint

and so on with the rest of the project ...

Now the stage fails because of pytest 😉

thank you @joerick , you can close the issue (or add some tips in your docs about windows execution on one line and /d in case of cd 😉 ).