Add Buffer Protocol support to int.from_bytes #132108
Labels
interpreter-core
(Objects, Python, Grammar, and Parser dirs)
performance
Performance or resource usage
type-feature
A feature request or enhancement
Comments
cmaloney
added a commit
to cmaloney/cpython
that referenced
this issue
Apr 5, 2025
Speed up conversion from `bytes-like` objects like `bytearray` while
keeping conversion from `bytes` stable.
On a `--with-lto --enable-optimizaitons` build on my 64 bit Linux box:
new:
from_bytes_flags: Mean +- std dev: 28.6 ns +- 0.5 ns
bench_convert[bytes]: Mean +- std dev: 50.4 ns +- 1.4 ns
bench_convert[bytearray]: Mean +- std dev: 51.3 ns +- 0.7 ns
old:
from_bytes_flags: Mean +- std dev: 28.1 ns +- 1.1 ns
bench_convert[bytes]: Mean +- std dev: 50.3 ns +- 4.3 ns
bench_convert[bytearray]: Mean +- std dev: 64.7 ns +- 0.9 ns
Benchmark code:
```python
import pyperf
import time
def from_bytes_flags(loops):
range_it = range(loops)
t0 = time.perf_counter()
for _ in range_it:
int.from_bytes(b'\x00\x10', byteorder='big')
int.from_bytes(b'\x00\x10', byteorder='little')
int.from_bytes(b'\xfc\x00', byteorder='big', signed=True)
int.from_bytes(b'\xfc\x00', byteorder='big', signed=False)
int.from_bytes([255, 0, 0], byteorder='big')
return time.perf_counter() - t0
sample_bytes = [
b'',
b'\x00',
b'\x01',
b'\x7f',
b'\x80',
b'\xff',
b'\x01\x00',
b'\x7f\xff',
b'\x80\x00',
b'\xff\xff',
b'\x01\x00\x00',
]
sample_bytearray = [bytearray(v) for v in sample_bytes]
def bench_convert(loops, values):
range_it = range(loops)
t0 = time.perf_counter()
for _ in range_it:
for val in values:
int.from_bytes(val)
return time.perf_counter() - t0
runner = pyperf.Runner()
runner.bench_time_func('from_bytes_flags', from_bytes_flags, inner_loops=10)
runner.bench_time_func('bench_convert[bytes]', bench_convert, sample_bytes, inner_loops=10)
runner.bench_time_func('bench_convert[bytearray]', bench_convert, sample_bytearray, inner_loops=10)
```
cmaloney
added a commit
to cmaloney/cpython
that referenced
this issue
Apr 5, 2025
Speed up conversion from `bytes-like` objects like `bytearray` while
keeping conversion from `bytes` stable.
On a `--with-lto --enable-optimizaitons` build on my 64 bit Linux box:
new:
from_bytes_flags: Mean +- std dev: 28.6 ns +- 0.5 ns
bench_convert[bytes]: Mean +- std dev: 50.4 ns +- 1.4 ns
bench_convert[bytearray]: Mean +- std dev: 51.3 ns +- 0.7 ns
old:
from_bytes_flags: Mean +- std dev: 28.1 ns +- 1.1 ns
bench_convert[bytes]: Mean +- std dev: 50.3 ns +- 4.3 ns
bench_convert[bytearray]: Mean +- std dev: 64.7 ns +- 0.9 ns
Benchmark code:
```python
import pyperf
import time
def from_bytes_flags(loops):
range_it = range(loops)
t0 = time.perf_counter()
for _ in range_it:
int.from_bytes(b'\x00\x10', byteorder='big')
int.from_bytes(b'\x00\x10', byteorder='little')
int.from_bytes(b'\xfc\x00', byteorder='big', signed=True)
int.from_bytes(b'\xfc\x00', byteorder='big', signed=False)
int.from_bytes([255, 0, 0], byteorder='big')
return time.perf_counter() - t0
sample_bytes = [
b'',
b'\x00',
b'\x01',
b'\x7f',
b'\x80',
b'\xff',
b'\x01\x00',
b'\x7f\xff',
b'\x80\x00',
b'\xff\xff',
b'\x01\x00\x00',
]
sample_bytearray = [bytearray(v) for v in sample_bytes]
def bench_convert(loops, values):
range_it = range(loops)
t0 = time.perf_counter()
for _ in range_it:
for val in values:
int.from_bytes(val)
return time.perf_counter() - t0
runner = pyperf.Runner()
runner.bench_time_func('from_bytes_flags', from_bytes_flags, inner_loops=10)
runner.bench_time_func('bench_convert[bytes]', bench_convert, sample_bytes, inner_loops=10)
runner.bench_time_func('bench_convert[bytearray]', bench_convert, sample_bytearray, inner_loops=10)
```
picnixz
added
interpreter-core
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Feature or enhancement
Proposal:
Currently
int.from_bytestakes aPyObject*and always converts it to abytesusingPyObject_Bytes. When a "bytes-like" object is passed in, such asbytearrayin_pylong, thePyObject_Bytesfunction has to make a newPyBytes*and copy the data from the bytes-like into it. Utilize the buffer protocol, with a fallback to PyObject_Bytes, to remove that object allocation and copy making the conversion faster.I think the fallback is needed because
PyObject_Bytessupports additional methods, such as.__bytes__()to construct thebytes, which should remain supported. That means need to keep doingbytes as bytes_obj: objectin Argument Clinic, rather thanbytes as bytes_obj: Py_buffer.Has this already been discussed elsewhere?
This is a minor feature, which does not need previous discussion elsewhere
Links to previous discussion of this feature:
No response
Linked PRs
The text was updated successfully, but these errors were encountered: