Skip to content

Document problems with multi-threading #1141

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 13, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 29 additions & 24 deletions docs/intro.rst
Original file line number Diff line number Diff line change
Expand Up @@ -6,75 +6,76 @@ system that mocks the Python file system modules.
Using pyfakefs, your tests operate on a fake file system in memory without touching the real disk.
The software under test requires no modification to work with pyfakefs.

pyfakefs works with CPython 3.7 and above, on Linux, Windows and macOS,
``pyfakefs`` works with CPython 3.7 and above, on Linux, Windows and macOS,
and with PyPy3.

pyfakefs works with `pytest <doc.pytest.org>`__ version 3.0.0 or above by
``pyfakefs`` works with `pytest <doc.pytest.org>`__ version 6.2.5 or above by
providing the `fs` fixture that enables the fake filesystem.

Installation
------------
pyfakefs is available on `PyPI <https://pypi.python.org/pypi/pyfakefs/>`__.
``pyfakefs`` is available on `PyPI <https://pypi.python.org/pypi/pyfakefs/>`__.
The latest released version can be installed from PyPI:

.. code:: bash

pip install pyfakefs

The latest main can be installed from the GitHub sources:
The latest development version (main branch) can be installed from the GitHub sources:

.. code:: bash

pip install git+https://github.com/pytest-dev/pyfakefs

Features
--------
- Code executed under pyfakefs works transparently on a memory-based file
- Code executed under ``pyfakefs`` works transparently on a memory-based file
system without the need of special commands. The same code that works on
the real filesystem will work on the fake filesystem if running under
pyfakefs.
``pyfakefs``.

- pyfakefs provides direct support for `pytest` (via the `fs` fixture)
- ``pyfakefs`` provides direct support for `pytest` (via the `fs` fixture)
and `unittest` (via a `TestCase` base class), but can also be used with
other test frameworks.

- Each pyfakefs test starts with an empty file system, but it is possible to
map files and directories from the real file system into the fake
- Each ``pyfakefs`` test starts with an empty (except for the :ref:`os_temporary_directories`) file system,
but it is possible to map files and directories from the real file system into the fake
filesystem if needed.

- No files in the real file system are changed during the tests, even in the
case of writing to mapped real files.

- pyfakefs keeps track of the filesystem size if configured. The file system
size can be configured arbitrarily.
- ``pyfakefs`` keeps track of the filesystem size if configured. The file system
size can be configured arbitrarily. It is also possible to create files with a defined
size without setting contents.

- it is possible to pause and resume using the fake filesystem, if the
real file system has to be used in a test step
- It is possible to pause and resume using the fake filesystem, if the
real file system has to be used in a test step.

- pyfakefs defaults to the OS it is running on, but can also be configured
- ``pyfakefs`` defaults to the OS it is running on, but can also be configured
to test code running under another OS (Linux, macOS or Windows).

- pyfakefs can be configured to behave as if running as a root or as a
- ``pyfakefs`` can be configured to behave as if running as a root or as a
non-root user, independently from the actual user.

.. _limitations:

Limitations
-----------
- pyfakefs will not work with Python libraries (other than `os` and `io`) that
- ``pyfakefs`` will not work with Python libraries (other than `os` and `io`) that
use C libraries to access the file system, because it cannot patch the
underlying C libraries' file access functions

- pyfakefs patches most kinds of importing file system modules automatically,
- ``pyfakefs`` patches most kinds of importing file system modules automatically,
but there are still some cases where this will not work.
See :ref:`customizing_patcher` for more information and ways to work around
this.

- pyfakefs does not retain the MRO for file objects, so you cannot rely on
- ``pyfakefs`` does not retain the MRO for file objects, so you cannot rely on
checks using `isinstance` for these objects (for example, to differentiate
between binary and textual file objects).

- pyfakefs is only tested with CPython and the newest PyPy versions, other
- ``pyfakefs`` is only tested with CPython and the newest PyPy versions, other
Python implementations will probably not work

- Differences in the behavior in different Linux distributions or different
Expand All @@ -84,26 +85,30 @@ Limitations
considered as reference systems. Additionally, the tests are run in Docker
containers with the latest CentOS, Debian, Fedora and Ubuntu images.

- pyfakefs may not work correctly if file system functions are patched by
- ``pyfakefs`` may not work correctly if file system functions are patched by
other means (e.g. using `unittest.mock.patch`) - see
:ref:`usage_with_mock_open` for more information

- pyfakefs will not work correctly with
- ``pyfakefs`` will not work correctly with
`behave <https://github.com/behave/behave>`__ due to the way it loads
the steps, if any filesystem modules are imported globally in the steps or
environment files; as a workaround, you may load them locally inside the
test steps (see `this issue <https://github.com/pytest-dev/pyfakefs/issues/703>`__)

- ``pyfakefs`` is not guaranteed to work correctly in multi-threading environments.
Specifically, it does not ensure concurrent write access to a file from different
threads, which is possible under Posix.

History
-------
pyfakefs was initially developed at Google by
``pyfakefs`` was initially developed at Google by
`Mike Bland <https://mike-bland.com/about.html>`__ as a modest
fake implementation of core Python modules. It was introduced to all of
Google in September 2006. Since then, it has been enhanced to extend its
functionality and usefulness. At last count, pyfakefs was used in over
functionality and usefulness. At last count, ``pyfakefs`` was used in over
20,000 Python tests at Google.

Google released pyfakefs to the public in 2011 as Google Code project
Google released ``pyfakefs`` to the public in 2011 as Google Code project
`pyfakefs <http://code.google.com/p/pyfakefs/>`__:

* Fork `jmcgeheeiv-pyfakefs <http://code.google.com/p/jmcgeheeiv-pyfakefs/>`__
Expand Down
16 changes: 14 additions & 2 deletions docs/troubleshooting.rst
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,16 @@ function correctly. Examples that have shown this problem include `GitPython`_
and `plumbum`_. Calling ``find_library`` also uses ``subprocess`` and does not work in
the fake filesystem.

Modules using multi-threading
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This includes for example the standard library module ``threading`` and the standard library
class ``concurrent.futures.ThreadPoolExecutor``.
Multi-threading is not specifically handled in ``pyfakefs`` and may lead to problems
in some cases. Specifically, writing to the same file from different threads (using
different file handles) may fail, while in the real filesystem this is possible in
Posix systems. There may be similar problems that we have not encountered yet, so
use ``pyfakefs`` in multi-threaded environments with care.

`sqlite3`_ (built-in)
~~~~~~~~~~~~~~~~~~~~~~~~
This is a database adapter written in C, which uses the database C API to access files.
Expand Down Expand Up @@ -132,7 +142,7 @@ the filesystem functions that ``pyfakefs`` patches, but in some cases it may
access other files in the packages. An example is the ``pytz`` module, which is loading timezone information
from configuration files. In these cases, you have to map the respective files
or directories from the real into the fake filesystem as described in
:ref:`real_fs_access`. For the timezone example, this could look like the following::
:ref:`real_fs_access`. For the timezone example, this could look like the following:

.. code:: python

Expand Down Expand Up @@ -172,6 +182,8 @@ Here's an example of how to add these using pytest:
)
return fs

.. _os_temporary_directories:

OS temporary directories
------------------------
Tests relying on a completely empty file system on test start will fail.
Expand Down Expand Up @@ -256,7 +268,7 @@ passed before the ``mocker`` fixture to ensure this:

Pathlib.Path objects created outside of tests
---------------------------------------------
An pattern which is more often seen with the increased usage of ``pathlib`` is the
A pattern which is seen more often with the increased usage of ``pathlib`` is the
creation of global ``pathlib.Path`` objects (instead of string paths) that are imported
into the tests. As these objects are created in the real filesystem,
they do not have the same attributes as fake ``pathlib.Path`` objects,
Expand Down
3 changes: 1 addition & 2 deletions docs/usage.rst
Original file line number Diff line number Diff line change
Expand Up @@ -115,8 +115,7 @@ method ``setUpClassPyfakefs`` instead:
self.assertTrue(os.path.exists(file_path))
.. note:: This feature cannot be used with a Python version before Python 3.8 due to
a missing feature in ``unittest``. If you use ``pytest`` for running tests using this feature,
you need to have at least ``pytest`` version 6.2 due to an issue in earlier versions.
a missing feature in ``unittest``.

.. caution:: If this is used, any changes made in the fake filesystem inside a test
will remain there for all following tests in the test class, if they are not reverted
Expand Down
Loading