Dropping concurrent.futures.Executor.map result cancels pending futures

[mirober]

Bug report

Bug description:

Code:

from concurrent.futures import ThreadPoolExecutor

data = [
    list(range(0, 5)),
    list(range(5, 10)),
]


def _f(x):
    print(f"Processing {x}")
    return True


print("=== 1 - No consumption from the iterator ===")

executor = ThreadPoolExecutor(max_workers=1)
for ints in data:
    executor.map(_f, ints)
executor.shutdown(wait=True)

print("=== 2 - Consume all values from the iterator ===")

executor = ThreadPoolExecutor(max_workers=1)
for ints in data:
    futures = executor.map(_f, ints)
    results = list(futures)
executor.shutdown(wait=True)

print("=== 3 - Consume one value from the iterator ===")

executor = ThreadPoolExecutor(max_workers=1)
for ints in data:
    futures = executor.map(_f, ints)
    first = next(futures)
executor.shutdown(wait=True)

print("=== 4 - Dropping iterator cancels remaining futures ===")

executor = ThreadPoolExecutor(max_workers=1)
futures = executor.map(_f, range(0, 5))
first = next(futures)
del futures
executor.shutdown(wait=True)

Result:

=== 1 - No consumption from the iterator ===
Processing 0
Processing 1
Processing 2
Processing 3
Processing 4
Processing 5
Processing 6
Processing 7
Processing 8
Processing 9
=== 2 - Consume all values from the iterator ===
Processing 0
Processing 1
Processing 2
Processing 3
Processing 4
Processing 5
Processing 6
Processing 7
Processing 8
Processing 9
=== 3 - Consume one value from the iterator ===
Processing 0
Processing 1
Processing 5
Processing 6
Processing 7
Processing 8
Processing 9
=== 4 - Dropping iterator cancels remaining futures ===
Processing 0
Processing 1

The behaviour seems to be:

• If the iterator returned from map is never used (case 1), futures are not cancelled
• If the iterator returned from map is exhausted (case 2), futures are not cancelled
• If the iterator returned from map is partially consumed and then dropped (cases 3 & 4), the remaining futures are cancelled

We hit this doing a version of case 3, calling any on the iterator, which short-circuited, causing the remaining futures to not execute. This tripped us up and seems like quite a confusing behaviour that is not flagged in the docs.

It looks like this is caused by this code: https://github.com/python/cpython/blob/main/Lib/concurrent/futures/_base.py#L669-L671

Possibly related to #108518

CPython versions tested on:

3.12

Operating systems tested on:

Linux