Switch terminal backend from Crossterm to Termina

[the-mikedavis]

Termina is a new terminal manipulation crate I've written that has a similar interface as Crossterm. The interface exposes lower-level details than Crossterm's, exposing escape sequences and pushing all of that handling to the application.

Termina has a minimal Windows integration that works because of ConPTY - we tell terminals on Windows to "speak VT" like a *NIX PTY would. This drops a lot of Windows-specific code for the old Console API and allows us to support VT features consistently on all platforms. In particular OSC52 bracketed paste should work on Windows now. The downside is that we will require Windows 10 like WezTerm: version 10.0.17763 or later. This is a fairly old version though, published ~Fall 2018. The Kitty keyboard protocol should work on WezTerm (when enabled in config) now, though it seems a bit buggy in my testing - needs some debugging.

Termina also has an interface for polling and reading for arbitrary events and escape sequences. (This exists in Crossterm too but is private.) This allows us to detect VT extensions like synchronized rendering and true color support while setting up the terminal.

Rendering should be smoother on this branch since synchronized output sequences are enabled when we detect support for them (fixes #731). Termina also has better "compression" for SGR sequences (things like foreground/background/underline colors, bold, italic, etc.) similar to crossterm-rs/crossterm#879 but covering all SGR attributes in addition to foreground and background - this should give the terminal less work to do when it parses the escapes we emit during rendering.

This needs some testing. I did some limited manual testing on Kitty and Ghostty on macOS, Kitty, Foot and WezTerm on Linux and cmd.exe, PowerShell and WezTerm on Windows.

Also fixes #12232 (the same way as #13223 did)

[RoloEdits]

I see you linked crossterm-rs/crossterm#879, would the performance gains seen there benefit Windows this time now that the terminal is "spoken" to the same way on all platforms?

[the-mikedavis]

Copying with the termcode (OSC52) provider randomly stops working

I was able to reproduce this with the same setup. I changed the OSC52 code a bit to do some explicit locking and flushing and I haven't been able to reproduce it so far. That part is a bit hacky - we basically print!() the command instead of executing it on the terminal buffer. We would need a bigger refactor to find a way to fix this properly.

I also noticed that true-color detection was broken for me in that workflow (i.e. SSH with Ghostty. Probably I would need to install the Ghostty terminfo on the NixOS side). I've updated the true-color detection to use a query following https://github.com/termstandard/colors#querying-the-terminal so it works out of the box in this case.

would the performance gains seen there benefit Windows this time...?

Crossterm does actually "speak VT," meaning that it emits escape sequences - like those used to set foreground/background colors - when it detects that ConPTY is available. The issue with its Windows handling is that it doesn't "hear VT": it enables ENABLE_VIRTUAL_TERMINAL_PROCESSING on the output handle (CONOUT$ or stdout) but not ENABLE_VIRTUAL_TERMINAL_INPUT on the input handle (CONIN$ or stdin) (Microsoft docs). So Crossterm was already using that optimization in crossterm-rs/crossterm#879 on Windows as well. It lacked the ability to read VT inputs from ConPTY though, so we wouldn't ever get a response to request/response type escape codes like DECRQSS/DECRPSS (what I was mentioning in #13224 (comment) and what makes the true color thing above possible) or detecting support for the Kitty keyboard protocol or synchronized outputs.

[RoloEdits]

Got some time to play around with this and I see some odd characters on the status bar (where I expect different ones) and when I press tab I see this:

Windows 11, WezTerm, Nushell

This was with:

config.enable_kitty_keyboard = false -- (also tried true and same thing)
config.allow_win32_input_mode = true-- (also tried false with kitty true and same thing)

2025-04-09T19:06:28.167 helix_tui::backend::termina [DEBUG] Detected terminal capabilities in 3.1107ms: Capabilities { kitty_keyboard: None, synchronized_output: true, true_color: false, extended_underlines: false }

The keys seem to work now, with the protocol enabled, where previously it didnt, but there is just this character glitch.

edit: actually when trying more keys, the arrow keys dont work. hjkl do still.

[chtenb]

On windows 11 it hangs during startup on wezterm, but launches correctly on windows terminal. The logs don't show anything.

[the-mikedavis]

The unicode stuff should be fixed in the latest push - I was accidentally resetting the input/output "code pages" (encodings) after querying for features. That should cover the weird glyphs for non-ascii characters and input problems like :q!. I'll spend some time on Windows today looking at the rest.

[the-mikedavis]

What does your WezTerm config look like?

[RoloEdits]

This is all with wezterm:

The text issue was fixed, but now no capabilities seem to have been detected

2025-04-10T12:39:55.065 helix_tui::backend::termina [DEBUG] Detected terminal capabilities in 3.7429ms: Capabilities { kitty_keyboard: None, synchronized_output: false, true_color: false, extended_underlines: false }

With allow_win32_input_mode set to false, ESC no longer works, like it did previously (with the text glitching).

For me, capslock works as expected. ctrl and alt also seem to be fine. Colors are fine.

[RoloEdits]

latest commit looks more promising with the capability check, though I wonder why true color and underlines are false?

2025-04-10T13:32:29.740 helix_tui::backend::termina [DEBUG] Detected terminal capabilities in 2.708ms: Capabilities { kitty_keyboard: Some, synchronized_output: true, true_color: false, extended_underlines: false }
2025-04-10T13:32:29.753 helix_tui::backend::termina [DEBUG] The terminal fully supports the requested keyboard enhancement flags

[the-mikedavis]

For better compatibility I switched to the foreground/background sequences suggested in the microsoft docs: https://learn.microsoft.com/en-us/windows/console/console-virtual-terminal-sequences#extended-colors. (As I understand it it's not spec-wise correct but is an idiom used in the wild more often than the correct syntax.) That should fix the colors on conhost and stable WezTerm (which is quite old, probably it bundles an old conhost). It's the same format as crossterm uses so it should be the same compatibility-wise.

That last push should fix the detection: out of the box WezTerm should be true for synchronized output and false or None for everything else (or Some if you have the kitty keyboard flag enabled). It does support true color and underlines but it doesn't currently respond to a request that would tell us if it did (that would be wezterm/wezterm#6856). Probably the log message should be clear that false actually means "no response to the query" rather than a definitive lack of support.

[RoloEdits]

I did notice some weird behavior with escaping and scrolling as well as just in general with escape, in that pressing it seems to cause some kind of delay. Or like the press doesnt actually "happen" until I give another input? Kind of hard to show this, but you can see me scrolling when you see the rapid change in key inputs at the bottom right. A menu also pops up when all I am doing is scrolling. Also when the prompt is open I am spamming escape but nothing happens until I press, in this instance, an arrow key:

which is quite old, probably it bundles an old conhost

That is correct, nightly now has one from this year, just a few weeks old I believe.

[the-mikedavis]

Is that with the Kitty keyboard protocol enabled? I've seen that behavior with WezTerm as well when it is enabled - termina doesn't seem to get anything from the input handle until another button is pressed - but I haven't debugged why yet. I don't see the same in Alacritty notably and I believe it has at least some support for the Kitty keyboard protocol.

[RoloEdits]

Is that with the Kitty keyboard protocol enabled?

That is correct, the behavior shown is with these set:

config.enable_kitty_keyboard = true
config.allow_win32_input_mode = true

[the-mikedavis]

It looks like this is a bug in the Kitty keyboard encoding function in WezTerm. I sent a patch wezterm/wezterm#6872. I'm not sure why the old encoding causes the key to 'queue up' and wait for a non-escape input but using the proper representation seems to fix it.

[RoloEdits]

Ok, added those wezterm patches and now running them.

2025-04-11T22:34:56.878 helix_tui::backend::termina [DEBUG] Detected terminal capabilities in 2.5524ms: Capabilities { kitty_keyboard: Some, synchronized_output: true, true_color: true, extended_underlines: true }
2025-04-11T22:34:56.889 helix_tui::backend::termina [DEBUG] The terminal fully supports the requested keyboard enhancement flags

All capabilities seem to be detected. The keys and mouse seem to work correctly. I will run this more and get back if anything is up.

Thanks for getting those upstream fixes in, the time and effort you spent is greatly appreciated.

[chtenb]

On windows 11 it hangs during startup on wezterm, but launches correctly on windows terminal. The logs don't show anything.

This seems to be resolved on the current tip.

[the-mikedavis]

That underline behavior may be a bug with WezTerm, I haven't been able to reproduce on Kitty, Alacritty or Foot yet. I'll take a look at how WezTerm is parsing those underline commands

[archseer]

I remember we were dealing with a wezterm bug like that before, I think it depended on the order of the style data

[archseer]

Previously seen here: #4061 (comment)

[the-mikedavis]

I believe that was a different issue with the crossterm backend at the time since I could reproduce it then on different terminals. I added a commit to work around the problem. It seems like WezTerm doesn't like having the underline color included in the big SGR update that updates all attributes that changed between two cells. We can work around it for now by emitting a separate SGR update whenever the underline color changes.

[j4james]

It seems like WezTerm doesn't like having the underline color included in the big SGR update that updates all attributes that changed between two cells.

This is likely because of a parameter limit. WezTerm has a limit of 16 (at least last I tested), and attributes with subparameters count as multiple parameters. For example an RGB color sequence like \e[38;2;255;128;64m counts as 5 parameters, even though only one attribute is being changed.

16 is actually a fairly common limit, and it's not even the lowest! I believe MLTerm has a limit of 10. That's just enough for two RGB colors, but if you're trying to change other attributes at the same time they're liable to get dropped or corrupted. So you need to be very careful how much you change in one sequence.

[the-mikedavis]

Aha, that makes sense, thanks! I'll update this escape to "chunk" the updates so they stay under that conservative limit. If we find lower limits we can tune that further.

[the-mikedavis]

Looks like WezTerm just bumped its CSI param limit from 32 to 256: wezterm/wezterm#6194. It's still probably better to keep the number of parameters low for wider compatibility though

[cotneit]

I accidentally ran
hx $nu.default-config-dir | path join 'zoxide.nu'
instead of
hx ($nu.default-config-dir | path join 'zoxide.nu')

And something broke (I'm not sure how this is supposed to work, but master seems to handle it)

This is with nushell 0.103.0 in Windows Terminal

master:

termina:
(Helix is running but output is nonsense, the only way to resolve this is to close the terminal tab)

Simplified example: hx | echo
master:

termina:
(Helix is running but output is nonsense, the only way to resolve this is to close the terminal tab)

[the-mikedavis]

That should be fixed now after helix-editor/termina@bd391b1. Also added the "chunking" for SGR attributes

[RoloEdits]

Checked to see if there is anything obvious in the samples for termina, and given that the edges are all pointers to the windows kernel(I believe), seems this is working great as a wrapper for stdio.

[mariocala]

My cursor is disappearing when I move from right to left over a closing bracket via the h key in normal mode.

This only happens with Ghostty+Tmux (not standalone Ghostty). Can't reproduce on (helix) main branch.

Screen.Recording.2025-05-19.at.22.14.22.mov

[RoloEdits]

For those building from source (and need to resolve Cargo.lock conflicts by restoring the changes here), given that there is no rev to lock to a commit, you need to add the .. pattern to WindowSize for the pixel info, as this is whats on master for termina.

Helix itself does not use them, as stated here helix-editor/termina#6 (review)

[mariocala]

My cursor is disappearing when I move from right to left over a closing bracket via the h key in normal mode.

This only happens with Ghostty+Tmux (not standalone Ghostty). Can't reproduce on (helix) main branch.

Screen.Recording.2025-05-19.at.22.14.22.mov

I think this has something to do with chunking in termina. Decreasing the parameter_chunk_size to 9 makes things more broken while increasing it to 11 fixes all apparent issues for me. Still not sure what the actual problem is though..