concurrent.futures Executor.map cancels other futures when one raises an exception

[KyleRAnderson]

Bug report

Checklist

• I am confident this is a bug in CPython, not a bug in a third-party project
• I have searched the CPython issue tracker,
  and am confident this bug has not been reported before

CPython versions tested on:

3.11

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.11.5 (main, Aug 24 2023, 12:23:19) [GCC 11.4.0]

A clear and concise description of the bug:

While using concurrent.futures.ProcessPoolExecutor.map when one of the calls to the function raises an exception, per the documentation, this exception is raised when the value is retrieved from the iterator returned by the call to concurrent.futures.ProcessPoolExecutor.map. However, it was seen that the iterator also terminates when the exception is retrieved from the iterator which is unexpected and doesn't align with what the built-in map function does. This means that other values fetched in separate calls cannot be retrieved with this method.

Reproducible example

import concurrent.futures
import contextlib

def add_one(value):
    if value == 2:
        raise ValueError()
    return value + 1

def test(executor_cm):
    with executor_cm as executor:
        values = executor.map(add_one, (1, 2, 3))
        values_it = iter(values)
        try:
            while True:
                try:
                    value = next(values_it)
                except ValueError:
                    print("value error")           
                else:
                    print(f"value: {value}")
        except StopIteration: pass

print("ProcessPoolExecutor")
test(concurrent.futures.ProcessPoolExecutor())
print("\nThreadPoolExecutor")
test(concurrent.futures.ThreadPoolExecutor())
print("\nBuilt-in map function")
class fake_executor:
    map = staticmethod(map)
test(contextlib.nullcontext(fake_executor))

The output of the example above:

ProcessPoolExecutor
value: 2
value error

ThreadPoolExecutor
value: 2
value error

Built-in map function
value: 2
value error
value: 4

The expected output for the example:

ProcessPoolExecutor
value: 2
value error
value: 4

ThreadPoolExecutor
value: 2
value error
value: 4

Built-in map function
value: 2
value error
value: 4

Linked PRs

• gh-108518: Make concurrent.futures.Executor.map() consistent with built-in map() #109497

[sterliakov]

Caused by

cpython/Lib/concurrent/futures/_base.py

Lines 612 to 625 in d08d49d

	def result_iterator():
	try:
	# reverse to keep finishing order
	fs.reverse()
	while fs:
	# Careful not to keep a reference to the popped future
	if timeout is None:
	yield _result_or_cancel(fs.pop())
	else:
	yield _result_or_cancel(fs.pop(), end_time - time.monotonic())
	finally:
	for future in fs:
	future.cancel()
	return result_iterator()

You cannot return to the generator function after raising any exception inside, and current behavior obviously contradicts the documentation. The problem applies to all supported python versions.

I can imagine the solution based on making result_iterator a class with __next__ method, however, this needs some insights from concurrent.futures experts (@gpshead) first: such changes will be breaking.