Unicode characters ≥ 0x10000 cannot be inputted/behaves unusually at the REPL terminal.

[haydenwong7bm]

Bug report

Bug description:

When the machine locale is set to UTF-8, when inputting a Unicode character ≥ 0x10000:
In CPython 3.13.5:
https://github.com/user-attachments/assets/7777b063-76fe-4929-b854-cae7d61807d2
In Cpython 3.14.0b4:

>>> Traceback (most recent call last):
  File "*\Python\Python314\Lib\_pyrepl\readline.py", line 394, in multiline_input
    return reader.readline()
           ~~~~~~~~~~~~~~~^^
  File "*\Python\Python314\Lib\_pyrepl\reader.py", line 748, in readline
    self.handle1()
    ~~~~~~~~~~~~^^
  File "*\Python\Python314\Lib\_pyrepl\reader.py", line 731, in handle1
    self.do_cmd(cmd)
    ~~~~~~~~~~~^^^^^
  File "*\Python\Python314\Lib\_pyrepl\reader.py", line 661, in do_cmd
    self.refresh()
    ~~~~~~~~~~~~^^
  File "*\Python\Python314\Lib\_pyrepl\reader.py", line 638, in refresh
    self.screen = self.calc_screen()
                  ~~~~~~~~~~~~~~~~^^
  File "*\Python\Python314\Lib\_pyrepl\completing_reader.py", line 261, in calc_screen
    screen = super().calc_screen()
  File "*\Python\Python314\Lib\_pyrepl\reader.py", line 315, in calc_screen
    colors = list(gen_colors(self.get_unicode()))
  File "*\Python\Python314\Lib\_pyrepl\utils.py", line 108, in gen_colors
    for color in gen_colors_from_token_stream(gen, line_lengths):
                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "*\Python\Python314\Lib\_pyrepl\utils.py", line 168, in gen_colors_from_token_stream
    for prev_token, token, next_token in token_window:
                                         ^^^^^^^^^^^^
  File "*\Python\Python314\Lib\_pyrepl\utils.py", line 363, in prev_next_window
    window = deque((None, next(iterator)), maxlen=3)
                          ~~~~^^^^^^^^^^
  File "*\Python\Python314\Lib\tokenize.py", line 582, in _generate_tokens_from_c_tokenizer
    for info in it:
                ^^
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 0-1: surrogates not allowed
>>> Traceback (most recent call last):
  File "*\Python\Python314\Lib\_pyrepl\readline.py", line 394, in multiline_input
    return reader.readline()
           ~~~~~~~~~~~~~~~^^
  File "*\Python\Python314\Lib\_pyrepl\reader.py", line 748, in readline
    self.handle1()
    ~~~~~~~~~~~~^^
  File "*\Python\Python314\Lib\_pyrepl\reader.py", line 731, in handle1
    self.do_cmd(cmd)
    ~~~~~~~~~~~^^^^^
  File "*\Python\Python314\Lib\_pyrepl\reader.py", line 661, in do_cmd
    self.refresh()
    ~~~~~~~~~~~~^^
  File "*\Python\Python314\Lib\_pyrepl\reader.py", line 638, in refresh
    self.screen = self.calc_screen()
                  ~~~~~~~~~~~~~~~~^^
  File "*\Python\Python314\Lib\_pyrepl\completing_reader.py", line 261, in calc_screen
    screen = super().calc_screen()
  File "*\Python\Python314\Lib\_pyrepl\reader.py", line 315, in calc_screen
    colors = list(gen_colors(self.get_unicode()))
  File "*\Python\Python314\Lib\_pyrepl\utils.py", line 108, in gen_colors
    for color in gen_colors_from_token_stream(gen, line_lengths):
                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "*\Python\Python314\Lib\_pyrepl\utils.py", line 168, in gen_colors_from_token_stream
    for prev_token, token, next_token in token_window:
                                         ^^^^^^^^^^^^
  File "*\Python\Python314\Lib\_pyrepl\utils.py", line 363, in prev_next_window
    window = deque((None, next(iterator)), maxlen=3)
                          ~~~~^^^^^^^^^^
  File "*\Python\Python314\Lib\tokenize.py", line 582, in _generate_tokens_from_c_tokenizer
    for info in it:
                ^^
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 0-1: surrogates not allowed

Two surrogates were "inputted" and so two UnicodeEncodeErrors.

CPython versions tested on:

3.13, 3.14

Operating systems tested on:

Windows

Linked PRs

• gh-136595: Normalize surrogate pairs in REPL input to fix UnicodeEnco… #136639