test failure in 5.8.0 when invoked with pytest

[jcfp]

Got around to updating the Debian package from 5.7.4 to 5.8.0 today, but unfortunately the tests fail the build when using pytest to run the testsuite. The problem does not occur with the other method of python3.13 -m pyfakefs.tests.all_tests: then 2686 tests are run without errors or failures.

Full build log here, the key part seems to be these 2 errors at setup of TestClassSetup.test_using_fakefs respectively TestClassSetup.test_using_fs_functions:

I: pybuild base:311: cd /builds/python-team/packages/python-pyfakefs/debian/output/source_dir/.pybuild/cpython3_3.13_pyfakefs/build; python3.13 -m pytest 
============================= test session starts ==============================
platform linux -- Python 3.13.2, pytest-8.3.5, pluggy-1.5.0
rootdir: /builds/python-team/packages/python-pyfakefs/debian/output/source_dir
configfile: pyproject.toml
plugins: pyfakefs-5.8.0, typeguard-4.4.2
collected 2709 items
pyfakefs/pytest_session_tests/test_patch_on_setup.py ...                 [  0%]
pyfakefs/pytest_tests/fake_fcntl_test.py .                               [  0%]
pyfakefs/pytest_tests/pytest_check_failed_plugin_test.py s               [  0%]
pyfakefs/pytest_tests/pytest_fixture_param_test.py XF.                   [  0%]
pyfakefs/pytest_tests/pytest_fixture_test.py xFF.                        [  0%]
pyfakefs/pytest_tests/pytest_module_fixture_test.py ..                   [  0%]
pyfakefs/pytest_tests/pytest_plugin_test.py .FF.......                   [  0%]
pyfakefs/pytest_tests/pytest_reload_pandas_test.py ss                    [  0%]
pyfakefs/pytest_tests/test_reload_local_import.py .F                     [  1%]
pyfakefs/tests/dynamic_patch_test.py .FF..                               [  1%]
pyfakefs/tests/example_test.py FFFFFFFs                                  [  1%]
pyfakefs/tests/fake_filesystem_glob_test.py .FFFFFFFF                    [  1%]
pyfakefs/tests/fake_filesystem_shutil_test.py .FFFFFFFFFF...F....s...sss [  2%]
ssssssssssssssssssssss.....F.FF.Fsssssssssss                             [  4%]
pyfakefs/tests/fake_filesystem_test.py ................................. [  5%]
........................................................................ [  8%]
................s..........s............................................ [ 10%]
...s.................................................................... [ 13%]
.F.FFFFFFFFFFF...FFFFF.FFFF..                                            [ 14%]
pyfakefs/tests/fake_filesystem_unittest_test.py .FFFF..F.FFFFFFFFFFFFFFF [ 15%]
FF..xFFFFFF.F.FFFFFFFFF.FFFFFFFFF.FFFFFFxF............F.FFF.FFFF....s.sE [ 18%]
E...FFF                                                                  [ 18%]
pyfakefs/tests/fake_filesystem_vs_real_test.py FF.F.FFF.FFF.FFFF..FFFFFF [ 19%]
FF.                                                                      [ 19%]
pyfakefs/tests/fake_legacy_modules_test.py ...............sFFFFsF....... [ 20%]
sF.ssF...FFFF.F.FF....F.FFFsFsFFFFFFFFsFFFF...F....FF.....F.....FF.....F [ 23%]
.....F........Fsssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 25%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 28%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 31%]
sssssssssssssssssssssssssssssssss                                        [ 32%]
pyfakefs/tests/fake_open_test.py ....................................... [ 33%]
................................ssssssssssssssssssssssssssssssssssssssss [ 36%]
sssssssssssssssssssssssssssssss.........sssssssss..s.ssss..sss.......... [ 39%]
...sssssssssssss..........ssssssssss..s.....ssssssss.....sssss.....sss.. [ 41%]
ss.....sssss..................ssssssssssssssssssF                        [ 43%]
pyfakefs/tests/fake_os_test.py ........................s................ [ 45%]
...................................s.................................... [ 47%]
........................................F............................... [ 50%]
........................................................................ [ 53%]
.........................................sssssssssssssssssssssssssssssss [ 55%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 58%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 61%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 63%]
sssssssssssssssssssssssssssssssssssssssssssssssssss..................... [ 66%]
.................................................................sssssss [ 69%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 71%]
sssssss.................s...................................s......sssss [ 74%]
ssssssssssssssssssssssssssssssssssssssssssss...........sssssssssss...... [ 77%]
..............ssssssssssssssssssss.....................sssssssssssssssss [ 79%]
...............sssssssssssssssss...............sssssssssssssssss........ [ 82%]
.......sssssssssssssssss.......FF.....F.F..sssssssssssss                 [ 84%]
pyfakefs/tests/fake_pathlib_test.py ..FFsFssssssssssssss.Fs......sssssss [ 85%]
ss........ssssssss........ssssssssF.FFFsFsFFFFFFFFsFFFFsssssssssssssssss [ 88%]
ssssF.F....sF.ssF...FFFF.F.FF....sssssssssssssssssssssssssssss..FF....FF [ 91%]
.....F.....FF..F.FF.....F...F....Fssssssssssssssssssssssssssssssssssssss [ 93%]
ssssssFF...s....................................................FFFF     [ 96%]
pyfakefs/tests/fake_stat_time_test.py .......sssssss........ssssssss.... [ 97%]
...sssssss........ssssssss.....sssss......ssssss                         [ 99%]
pyfakefs/tests/fake_tempfile_test.py .F......                            [ 99%]
pyfakefs/tests/mox3_stubout_test.py .........                            [ 99%]
pyfakefs/tests/patched_packages_test.py .F                               [100%]
==================================== ERRORS ====================================
______________ ERROR at setup of TestClassSetup.test_using_fakefs ______________
cls = <class 'pyfakefs.tests.fake_filesystem_unittest_test.TestClassSetup'>
    @classmethod
    def setUpClass(cls):
        cls.setUpClassPyfakefs()
>       cls.fake_fs().create_file("foo/bar", contents="test")
/builds/python-team/packages/python-pyfakefs/debian/output/source_dir/.pybuild/cpython3_3.13_pyfakefs/build/pyfakefs/tests/fake_filesystem_unittest_test.py:951: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/builds/python-team/packages/python-pyfakefs/debian/output/source_dir/.pybuild/cpython3_3.13_pyfakefs/build/pyfakefs/fake_filesystem.py:2295: in create_file
    return self.create_file_internally(
/builds/python-team/packages/python-pyfakefs/debian/output/source_dir/.pybuild/cpython3_3.13_pyfakefs/build/pyfakefs/fake_filesystem.py:2571: in create_file_internally
    self.raise_os_error(errno.EEXIST, path)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <pyfakefs.fake_filesystem.FakeFilesystem object at 0x7f714281d010>
err_no = 17, filename = 'C:\\foo\\bar', winerror = None
    def raise_os_error(
        self,
        err_no: int,
        filename: Optional[AnyString] = None,
        winerror: Optional[int] = None,
    ) -> NoReturn:
        """Raises OSError.
        The error message is constructed from the given error code and shall
        start with the error string issued in the real system.
        Note: this is not true under Windows if winerror is given - in this
        case a localized message specific to winerror will be shown in the
        real file system.
    
        Args:
            err_no: A numeric error code from the C variable errno.
            filename: The name of the affected file, if any.
            winerror: Windows only - the specific Windows error code.
        """
        message = os.strerror(err_no) + " in the fake filesystem"
        if winerror is not None and sys.platform == "win32" and self.is_windows_fs:
            raise OSError(err_no, message, filename, winerror)
>       raise OSError(err_no, message, filename)
E       FileExistsError: [Errno 17] File exists in the fake filesystem: 'C:\\foo\\bar'
/builds/python-team/packages/python-pyfakefs/debian/output/source_dir/.pybuild/cpython3_3.13_pyfakefs/build/pyfakefs/fake_filesystem.py:492: FileExistsError
___________ ERROR at setup of TestClassSetup.test_using_fs_functions ___________
cls = <class 'pyfakefs.tests.fake_filesystem_unittest_test.TestClassSetup'>
    @classmethod
    def setUpClass(cls):
        cls.setUpClassPyfakefs()
>       cls.fake_fs().create_file("foo/bar", contents="test")
/builds/python-team/packages/python-pyfakefs/debian/output/source_dir/.pybuild/cpython3_3.13_pyfakefs/build/pyfakefs/tests/fake_filesystem_unittest_test.py:951: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/builds/python-team/packages/python-pyfakefs/debian/output/source_dir/.pybuild/cpython3_3.13_pyfakefs/build/pyfakefs/fake_filesystem.py:2295: in create_file
    return self.create_file_internally(
/builds/python-team/packages/python-pyfakefs/debian/output/source_dir/.pybuild/cpython3_3.13_pyfakefs/build/pyfakefs/fake_filesystem.py:2571: in create_file_internally
    self.raise_os_error(errno.EEXIST, path)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <pyfakefs.fake_filesystem.FakeFilesystem object at 0x7f714281d010>
err_no = 17, filename = 'C:\\foo\\bar', winerror = None
    def raise_os_error(
        self,
        err_no: int,
        filename: Optional[AnyString] = None,
        winerror: Optional[int] = None,
    ) -> NoReturn:
        """Raises OSError.
        The error message is constructed from the given error code and shall
        start with the error string issued in the real system.
        Note: this is not true under Windows if winerror is given - in this
        case a localized message specific to winerror will be shown in the
        real file system.
    
        Args:
            err_no: A numeric error code from the C variable errno.
            filename: The name of the affected file, if any.
            winerror: Windows only - the specific Windows error code.
        """
        message = os.strerror(err_no) + " in the fake filesystem"
        if winerror is not None and sys.platform == "win32" and self.is_windows_fs:
            raise OSError(err_no, message, filename, winerror)
>       raise OSError(err_no, message, filename)
E       FileExistsError: [Errno 17] File exists in the fake filesystem: 'C:\\foo\\bar'
/builds/python-team/packages/python-pyfakefs/debian/output/source_dir/.pybuild/cpython3_3.13_pyfakefs/build/pyfakefs/fake_filesystem.py:492: FileExistsError
=================================== FAILURES ===================================
__________ test_example_file_passing_using_parametrized_fixture[fs0] ___________
fs = <pyfakefs.fake_filesystem.FakeFilesystem object at 0x7f714281d010>
    @pytest.mark.parametrize("fs", [[None, [example]]], indirect=True)
    def test_example_file_passing_using_parametrized_fixture(fs):
        """Test passes if using a fixture that reloads the module containing
        EXAMPLE_FILE"""
>       fs.create_file(example.EXAMPLE_FILE, contents="stuff here")
/builds/python-team/packages/python-pyfakefs/debian/output/source_dir/.pybuild/cpython3_3.13_pyfakefs/build/pyfakefs/pytest_tests/pytest_fixture_param_test.py:33: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/builds/python-team/packages/python-pyfakefs/debian/output/source_dir/.pybuild/cpython3_3.13_pyfakefs/build/pyfakefs/fake_filesystem.py:2295: in create_file
    return self.create_file_internally(
/builds/python-team/packages/python-pyfakefs/debian/output/source_dir/.pybuild/cpython3_3.13_pyfakefs/build/pyfakefs/fake_filesystem.py:2571: in create_file_internally
    self.raise_os_error(errno.EEXIST, path)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <pyfakefs.fake_filesystem.FakeFilesystem object at 0x7f714281d010>
err_no = 17, filename = '/test/file', winerror = None
    def raise_os_error(
        self,
        err_no: int,
        filename: Optional[AnyString] = None,
        winerror: Optional[int] = None,
    ) -> NoReturn:
        """Raises OSError.
        The error message is constructed from the given error code and shall
        start with the error string issued in the real system.
        Note: this is not true under Windows if winerror is given - in this
        case a localized message specific to winerror will be shown in the
        real file system.
    
        Args:
            err_no: A numeric error code from the C variable errno.
            filename: The name of the affected file, if any.
            winerror: Windows only - the specific Windows error code.
        """
        message = os.strerror(err_no) + " in the fake filesystem"
        if winerror is not None and sys.platform == "win32" and self.is_windows_fs:
            raise OSError(err_no, message, filename, winerror)
>       raise OSError(err_no, message, filename)
E       FileExistsError: [Errno 17] File exists in the fake filesystem: '/test/file'
/builds/python-team/packages/python-pyfakefs/debian/output/source_dir/.pybuild/cpython3_3.13_pyfakefs/build/pyfakefs/fake_filesystem.py:492: FileExistsError

[more of the same...]

[mrbean-bremen]

Thanks - that's unfortunate! Currently, I've no idea where this comes from, will have to check later.

The CI actually uses pytest pyfakefs/pytest_tests, which passes for all tested Python versions and systems (e.g. Ubuntu, macOS and Windows).
There are also tests running in a few docker containers (including debian), but they do not use pytest so far - probably makes sense to change this.

[jcfp]

Just tried to run the tests in each subdirectory separately with pytest (i.e. python3 -m pytest pyfakefs/<subdir>/ once per subdir) and then everything completes just fine, regardless of order.

Combining things into a single command with all 3 subdirs as arguments at once (python3 -m pytest pyfakefs/pytest_tests/ pyfakefs/tests/ pyfakefs/pytest_session_tests/), outcome differs depending on the order of the directory arguments: failures happen unless "pytest_session_tests" comes last.

[mrbean-bremen]

Ok, there is at least one problem: the session tests (actually only one test) have to be run separately, because they use a session fixture. This is also done in the CI, forgot to mention that.

[mrbean-bremen]

This also explains it: I've added the session fixture test in the latest update. Not sure what best to do about this - running all tests in the same session is indeed not expected to work.

[jcfp]

For the Debian package I'll just pass all 3 subdirs as arguments, to ensure tests are run in the correct order. I don't know enough about manipulating collection and/or test ordering in pytest to suggest a nicer solution so that python3 -m pytest just works as before.

[mrbean-bremen]

I will see if there is a better way for this - maybe it is not really needed to use a session fixture in the test. If a module fixture is sufficient to reproduce the problem that it tests, I will change it, and we wouldn't have the problem anymore.
I understand that this is not urgent for you, as you have found a workaround for now, so a patch release is not that important - correct me, if I'm wrong.

[mrbean-bremen]

Should be fixed in main now (using a module-scoped fixture was indeed sufficient).
Let me know if you need a patch release.

[jcfp]

No need for that on my end, already uploaded 5.8.0 with the workaround added and all tests are passing. As always, thanks for the quick responses and fix!