GH-132439: Fix REPL swallowing characters entered with AltGr on cmd.exe #132440

chris-eibl · 2025-04-12T14:31:29Z

E.g. on my keyboard with German layout, { is usually entered via pressing the AltGr key and 7, i.e. AltGr+7.

Likewise, }, [, ], \ and some more can only be entered via AltGr.

But since #128388 / #128389 these are swallowed by the REPL on Windows and can no longer be entered.

This happens in legacy Windows terminals, where the virtual terminal mode is turned off (e.g. cmd.exe).

In virtual terminal mode there are other issues, see #131878.

Issue: New REPL on Windows swallows characters entered via AltGr #132439

Lib/_pyrepl/windows_console.py

chris-eibl · 2025-04-12T14:42:29Z

Lib/_pyrepl/windows_console.py

                if block:
                    continue

                return None
            elif self.__vt_support:
                # If virtual terminal is enabled, scanning VT sequences
-                self.event_queue.push(rec.Event.KeyEvent.uChar.UnicodeChar)
+                self.event_queue.push(raw_key)


A missed opportunity when main was merged during development of #124119.

Which @sergey-miryanov takes care of here as part of #131901.

So maybe I should remove this one, but I'd really like to keep the other two raw_key changes (not only because I'd have to adapt almost all tests :)

IMHO you should merge my branch here :)

I can undo the change - then you won't have conflicts. I don't think we should (partially) merge between our two PRs?

Yeah, I think you are right - we shouldn't merge our PRs. You can keep your changes - I'm OK if here will be conflict.

chris-eibl · 2025-04-12T14:46:39Z

Lib/_pyrepl/windows_console.py

                        return Event(evt="key", data="\033")  # keymap.py uses this for meta
-                    return Event(evt="key", data=key, raw=key)
+                    return Event(evt="key", data=key, raw=raw_key)


Looking at the diff , previously

return Event( evt="key", data=code, raw=rec.Event.KeyEvent.uChar.UnicodeChar )

was used, which in the new code should have been raw_key?

Actually, I was wondering why this change did not break anything, but AFAICT the raw member is not used in the whole code base except

cpython/Lib/_pyrepl/unix_console.py

Line 513 in 5d8e432

def getpending(self):

where the two getpending implementations dutifully respect e.raw += e.raw :)

When searching in the code base for getpending I found

cpython/Lib/_pyrepl/windows_console.py

Lines 518 to 521 in 5d8e432

def getpending(self) -> Event:

"""Return the characters that have been typed but not yet

processed."""

return Event("key", "", b"")

which clearly seems to be a bug to me? Because even though WindowsConsole._read_input() only reads one Windows INPUT_RECORD per call, get_event could have put something into self.event_queue.

I've addressed this in https://github.com/python/cpython/pull/132889/files#r2059235071.

Lib/_pyrepl/windows_console.py

chris-eibl · 2025-04-12T14:53:03Z

Adding @vstinner, @eendebakpt and @paulie4 , since they were involved in #128389 and already talked about AltGr.

chris-eibl · 2025-04-12T14:56:22Z

PS: switching to an english keyboard layout, I can use the AltGr key like right-Alt, since it is not used in this keyboard layout :)

chris-eibl · 2025-04-12T17:15:46Z

The failing of Ubuntu (free-threading) is definitely unrelated, since this is a Windows specific change.

tomasr8 · 2025-04-17T19:21:45Z

Do you think it'd be possible to add some tests for this change to ensure we don't accidentally regress again?

chris-eibl · 2025-04-17T19:43:44Z

Yeah, I've already thought about it, but am still unsure how to best do it.

ATM, I am thinking of mocking

cpython/Lib/_pyrepl/windows_console.py

Line 411 in b87189d

def _read_input(self, block: bool = True) -> INPUT_RECORD | None:

to be able to test

cpython/Lib/_pyrepl/windows_console.py

Lines 426 to 432 in b87189d

    
               def get_event(self, block: bool = True) -> Event | None: 
        
                   """Return an Event instance.  Returns None if |block| is false 
        
                   and there is no event pending, otherwise waits for the 
        
                   completion of an event.""" 
        
                   while self.event_queue.empty(): 
        
                       rec = self._read_input(block)

AFAICT, all the existing tests just mock get_event, but for this, IMHO get_event itself must be tested.
Mocking _read_input seems to be the best way forward.

WDYT?

tomasr8 · 2025-04-17T20:17:22Z

Agreed that we shouldn't be mocking get_event since that's where we're making changes. _read_input seems like a good option, we can capture some real inputs and then replay them with it.

Lib/test/test_pyrepl/test_windows_console.py

Misc/NEWS.d/next/Library/2025-04-12-16-29-42.gh-issue-132439.3twrU6.rst

Lib/test/test_pyrepl/test_windows_console.py

Co-authored-by: Stan Ulbrych <[email protected]>

because it will be part of 131901, anywayy

because it is unused

Lib/test/test_pyrepl/test_windows_console.py

ambv

Excellent work again! Very well thought through.

miss-islington-app · 2025-05-05T16:45:48Z

Thanks @chris-eibl for the PR, and @ambv for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

miss-islington-app · 2025-05-05T16:45:51Z

Sorry, @chris-eibl and @ambv, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 07f416a3f063db6b91b8b99ff61a51b64b0503f1 3.13

sergey-miryanov · 2025-05-05T17:12:53Z

Will repeat here too - problem with backport because VT support was not backported #130805 (comment) (for completeness)

ambv · 2025-05-05T17:32:09Z

Yeah, I am backporting the VT support too because it's the only way we can have a sensibly similar codebase for bug fixes.

…ltGr on cmd.exe (pythonGH-132440) (cherry picked from commit 07f416a) Co-authored-by: Chris Eibl <[email protected]> Co-authored-by: Stan Ulbrych <[email protected]>

bedevere-app · 2025-05-05T17:46:04Z

GH-133460 is a backport of this pull request to the 3.13 branch.

…n cmd.exe (GH-132440) (GH-133460) (cherry picked from commit 07f416a) Co-authored-by: Chris Eibl <[email protected]> Co-authored-by: Stan Ulbrych <[email protected]>

chris-eibl added 2 commits April 12, 2025 16:01

do not swallow characters that have been entered via AltGr

59d71b0

blurb it

cca6155

chris-eibl requested review from pablogsal, lysnikolaou and ambv as code owners April 12, 2025 14:31

bedevere-app bot added the awaiting review label Apr 12, 2025

bedevere-app bot mentioned this pull request Apr 12, 2025

New REPL on Windows swallows characters entered via AltGr #132439

Closed

chris-eibl commented Apr 12, 2025

View reviewed changes

Lib/_pyrepl/windows_console.py Show resolved Hide resolved

chris-eibl added OS-windows 3.13 bugs and security fixes 3.14 bugs and security fixes topic-repl Related to the interactive shell labels Apr 12, 2025

unrelated stuff worth to do (?)

1e61540

chris-eibl commented Apr 12, 2025

View reviewed changes

Lib/_pyrepl/windows_console.py Outdated Show resolved Hide resolved

chris-eibl requested a review from vstinner April 12, 2025 14:49

picnixz added needs backport to 3.13 bugs and security fixes and removed 3.13 bugs and security fixes 3.14 bugs and security fixes labels Apr 12, 2025

chris-eibl added 3 commits April 19, 2025 08:30

Merge remote-tracking branch 'origin/main' into fix_altgr

c60a727

fix missing con.restore() in test_multiline_ctrl_z

ed035e5

add WindowsConsoleGetEventTests

69e8779

chris-eibl commented Apr 20, 2025

View reviewed changes

Lib/test/test_pyrepl/test_windows_console.py Show resolved Hide resolved

bedevere-app bot added the awaiting core review label Apr 21, 2025

StanFromIreland reviewed Apr 22, 2025

View reviewed changes

Misc/NEWS.d/next/Library/2025-04-12-16-29-42.gh-issue-132439.3twrU6.rst Outdated Show resolved Hide resolved

sergey-miryanov mentioned this pull request Apr 24, 2025

Brackets "[" and "]" writing or inserting does not work in Python 3.13 interactive shell (Portuguese keyboard) #132871

Closed

This was referenced Apr 24, 2025

Pasting in new REPL is slow #130328

Closed

GH-130328: pasting in new REPL is slow on Windows #132884

Merged

chris-eibl commented Apr 25, 2025

View reviewed changes

Lib/test/test_pyrepl/test_windows_console.py Outdated Show resolved Hide resolved

chris-eibl and others added 6 commits April 25, 2025 21:40

Apply suggestions from code review

e67452b

Co-authored-by: Stan Ulbrych <[email protected]>

Update Lib/test/test_pyrepl/test_windows_console.py

396f48e

Merge branch 'main' into fix_altgr

40b339a

revert change in virtual terminal path

1fbe240

because it will be part of 131901, anywayy

Merge remote-tracking branch 'origin/main' into fix_altgr

2d59563

remove raw part of Event in get_event() in case of windows_console

adb89ba

because it is unused

chris-eibl commented May 5, 2025

View reviewed changes

Lib/test/test_pyrepl/test_windows_console.py Show resolved Hide resolved

ambv changed the title ~~GH-132439: REPL on Windows swallows characters entered via AltGr~~ GH-132439: Fix REPL swallowing characters entered with AltGr May 5, 2025

ambv approved these changes May 5, 2025

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting core review labels May 5, 2025

ambv merged commit 07f416a into python:main May 5, 2025
52 checks passed

bedevere-app bot removed the awaiting merge label May 5, 2025

miss-islington-app bot assigned ambv May 5, 2025

ambv changed the title ~~GH-132439: Fix REPL swallowing characters entered with AltGr~~ GH-132439: Fix REPL swallowing characters entered with AltGr on cmd.exe May 5, 2025

bedevere-app bot removed the needs backport to 3.13 bugs and security fixes label May 5, 2025

chris-eibl deleted the fix_altgr branch May 5, 2025 18:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-132439: Fix REPL swallowing characters entered with AltGr on cmd.exe #132440

GH-132439: Fix REPL swallowing characters entered with AltGr on cmd.exe #132440

chris-eibl commented Apr 12, 2025 •

edited

Loading

chris-eibl Apr 12, 2025

chris-eibl Apr 20, 2025

sergey-miryanov Apr 20, 2025 •

edited

Loading

chris-eibl Apr 20, 2025

sergey-miryanov Apr 21, 2025

chris-eibl Apr 12, 2025

chris-eibl Apr 20, 2025

chris-eibl Apr 20, 2025 •

edited

Loading

chris-eibl commented Apr 12, 2025

chris-eibl commented Apr 12, 2025

chris-eibl commented Apr 12, 2025

tomasr8 commented Apr 17, 2025

chris-eibl commented Apr 17, 2025

tomasr8 commented Apr 17, 2025

ambv left a comment

miss-islington-app bot commented May 5, 2025

miss-islington-app bot commented May 5, 2025

sergey-miryanov commented May 5, 2025

ambv commented May 5, 2025

bedevere-app bot commented May 5, 2025

	def getpending(self) -> Event:
	"""Return the characters that have been typed but not yet
	processed."""
	return Event("key", "", b"")

GH-132439: Fix REPL swallowing characters entered with AltGr on cmd.exe #132440

GH-132439: Fix REPL swallowing characters entered with AltGr on cmd.exe #132440

Conversation

chris-eibl commented Apr 12, 2025 • edited Loading

chris-eibl Apr 12, 2025

Choose a reason for hiding this comment

chris-eibl Apr 20, 2025

Choose a reason for hiding this comment

sergey-miryanov Apr 20, 2025 • edited Loading

Choose a reason for hiding this comment

chris-eibl Apr 20, 2025

Choose a reason for hiding this comment

sergey-miryanov Apr 21, 2025

Choose a reason for hiding this comment

chris-eibl Apr 12, 2025

Choose a reason for hiding this comment

chris-eibl Apr 20, 2025

Choose a reason for hiding this comment

chris-eibl Apr 20, 2025 • edited Loading

Choose a reason for hiding this comment

chris-eibl commented Apr 12, 2025

chris-eibl commented Apr 12, 2025

chris-eibl commented Apr 12, 2025

tomasr8 commented Apr 17, 2025

chris-eibl commented Apr 17, 2025

tomasr8 commented Apr 17, 2025

ambv left a comment

Choose a reason for hiding this comment

miss-islington-app bot commented May 5, 2025

miss-islington-app bot commented May 5, 2025

sergey-miryanov commented May 5, 2025

ambv commented May 5, 2025

bedevere-app bot commented May 5, 2025

chris-eibl commented Apr 12, 2025 •

edited

Loading

sergey-miryanov Apr 20, 2025 •

edited

Loading

chris-eibl Apr 20, 2025 •

edited

Loading