Skip to content

gh-129005: Align FileIO.readall between _pyio and _io #129705

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Feb 7, 2025
Merged

gh-129005: Align FileIO.readall between _pyio and _io #129705

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
50389df
gh-12005: Align FileIO.readall between _pyio and _io
cmaloney Feb 5, 2025
4520ecc
Update Lib/_pyio.py
cmaloney Feb 5, 2025
6c3ac57
Use len(result) rather than bufsize, _new_buffersize, make in terms o…
cmaloney Feb 6, 2025
0e15dee
Merge branch 'main' into fileio_readall
cmaloney Feb 6, 2025
b50fb66
Simplify control structure
cmaloney Feb 7, 2025
4da4e31
Fix whitespace
cmaloney Feb 7, 2025
09b52ed
Merge branch 'main' into fileio_readall
cmaloney Feb 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 19 additions & 10 deletions Lib/_pyio.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1674,22 +1674,31 @@ def readall(self):
except OSError:
pass

result = bytearray()
result = bytearray(bufsize)
bytes_read = 0
while True:
if len(result) >= bufsize:
bufsize = len(result)
bufsize += max(bufsize, DEFAULT_BUFFER_SIZE)
n = bufsize - len(result)
if bytes_read >= bufsize:
# Parallels _io/fileio.c new_buffersize
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the C code, new_buffersize() argument is bytes_read, not bufsize. You may keep new_buffersize() as a private module-level function.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the loop to no longer use bufsize at all, this is the only line that used it, and it feels more Pythonic to me to just use len(result).

That enables rewriting to:

        try:
            # Read until EOF (n == 0)
            while n := os.readinto(self._fd, memoryview(result)[bytes_read:]):
                bytes_read += n
                if bytes_read >= len(result):
                    result.resize(_new_buffersize(bytes_read))
        except BlockingIOError:
            if not bytes_read:
                return None
        assert len(result) - bytes_read >= 1, \
            "os.readinto buffer size 0 will result in erroneous EOF / returns 0"
        result.resize(bytes_read)
        return bytes(result)

which feels cleaner, but also starts changing structure relative to _io version.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

decided to refactor to this. Control flow feels a lot simpler to me and a lot more readable than the branches and breaks.

if bufsize > 65536:
addend = bufsize >> 3
else:
addend = 256 + bufsize
if addend < DEFAULT_BUFFER_SIZE:
addend = DEFAULT_BUFFER_SIZE
bufsize += addend
result.resize(bufsize)

assert len(result) - bytes_read >= 1, "Must read at least one byte"
try:
chunk = os.read(self._fd, n)
n = os.readinto(self._fd, memoryview(result)[bytes_read:])
except BlockingIOError:
if result:
if bytes_read:
break
return None
if not chunk: # reached the end of the file
if n == 0: # Reached the end of the file
break
result += chunk

bytes_read += n
result.resize(bytes_read)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

result = memoryview(result)[bytes_read:] would avoid a truncation which can imply a memory copy in the worst case, no?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the resize "shrink" in bytearray doesn't actually resize unless the buffer's "capacity" is 2x the requested size (https://github.com/python/cpython/blob/main/Objects/bytearrayobject.c#L201-L214). Just updates its internal "this is how long the bytes is" counter (which for things like full-file readall with known size, this should already be just one byte over the right size).

My plan currently is to make it so bytes(bytearray(10)) and bytearray(b'\0' * 10) both don't copy (Ongoing discussion in https://discuss.python.org/t/add-zero-copy-conversion-of-bytearray-to-bytes-by-providing-bytes/79164). Having a memoryview would mean there's more than one reference to the bytearray, and I couldn't do / use that optimization.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I'm fine with using result.resize() here.

return bytes(result)

def readinto(self, buffer):
Expand Down
2 changes: 2 additions & 0 deletions Misc/NEWS.d/next/Library/2025-02-05-13-19-15.gh-issue-129005.Sb69L_.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
``_pyio.FileIO.readall()`` now allocates, resizes, and fills a data buffer
using the same algorithm ``_io.FileIO.readall()`` uses.
Loading