Staying in normal mode after loosing focus

[mickael-menu]

First of all, thank you for this outstanding piece of software. This is the most promising system-wide Vim implementation I've tried so far. I'm using it mostly to replace Xcode 13's subpar Vim implementation now that they killed XVim.

There's one issue that keep tripping me up though: the Normal mode is lost regularly and I don't realise it until after I typed a few characters. For example:

• When switching to a different window, and coming back.
• When clicking (with the mouse, ew) on a different caret location.
• When triggering a passthrough keyboard shortcut (e.g. Cmd-S which I keep hitting so much it might be pathological).

Here's a live action movie of the issue:

lost-normal.mov

Ideally, in all these scenarii I would like kindaVim to stay (or at least restore) the Normal mode. But if at least the current selection (selected character simulating the block caret of Vim normal mode) was discarded when switching to Insert mode implicitly, I think the visual feedback would help.

Not sure if that would be useful, but the kAXFocusedUIElementChangedNotification and AXObserverAddNotification seem relevant for the issue when switching to a different window / UI control.

Thank you!

[godbout]

First of all, thank you for this outstanding piece of software.

thank you! took a while to reach a proper stage. but it was fun using kV while building kV in Xcode 😅️

This is the most promising system-wide Vim implementation I've tried so far. I'm using it mostly to replace Xcode 13's subpar Vim implementation now that they killed XVim.

yeah if i remember well Xcode announced Vim some months before i released kV 😅️ but they had less motions than kV. was hoping they would keep pushing but seems they're dropping it or what?

There's one issue that keep tripping me up though: the Normal mode is lost regularly and I don't realise it until after I typed a few characters. For example:

• When switching to a different window, and coming back.
• When clicking (with the mouse, ew) on a different caret location.
• When triggering a passthrough keyboard shortcut (e.g. Cmd-S which I keep hitting so much it might be pathological).

Here's a live action movie of the issue:

lost-normal.mov

yes. i think there's an easier way to describe this: every time the current UI Element loses focus, kV goes back to Insert Mode. see below for more information about why.

Ideally, in all these scenarii I would like kindaVim to stay (or at least restore) the Normal mode.

the problem is there's no macOS API to track a UI Element. you can track apps, you could track windows, but you can't track UI Elements. so it's not possible for kV to know when you are back to a previous element.

But if at least the current selection (selected character simulating the block caret of Vim normal mode) was discarded when switching to Insert mode implicitly, I think the visual feedback would help.

yes, so there's some issues with that too. the first one is that you can't detect through the AX API that you will leave a UI Element. you can be notified (like with the API that you're pointing below) ONCE the UI Element changed. again, because you can't track what was the previous one, you can't go back and discard the selection. and same, because you're notified ONLY after you've already left the UI Element, you can't make the change before leaving. a second reason is more deliberate. many users want to run programs (like Alfred's Universal Action) on selected text. this is where the passthrough come from. that allows the users to get back to Insert Mode/macOS and run stuff on the selected text. if you'd remove the block cursor/selection, then you'll not be able to apply anything on your kV Visual Mode selection.

Not sure if that would be useful, but the kAXFocusedUIElementChangedNotification and AXObserverAddNotification seem relevant for the issue when switching to a different window / UI control.

i'm actually using those APIs and something related in order to deliberately switch back to Insert Mode 😂️ like for example when you click somewhere. there's a possibility to do stuff, like when you click to a new UI Element (Text only?), then you restart the kV Engine and you're in Normal Mode directly in the new UI Element. (although you would lose all data between UI Elements. to do its job kV needs to grab A LOT of data from a UI Element, most of which are not transferrable.) but would you always want that? i remember playing with this at the beginning, but i've found the experience completely inconsistent. what worked best, with my experience, is to go back to Insert Mode as soon as you leave the current UI Element. maybe i just got used to it? personally it feels very natural to me. (almost maybe i have my escape key mapped to caps lock, and the whole thing is just smooth and natural.)

Thank you!

my pleasure. feel free to share more, ping me, send questions, argue. i'm here everyday. and it helps a lot, making me think, review mindsets and ideas, and ultimately can make kV better. so thank you!

[mickael-menu]

yeah if i remember well Xcode announced Vim some months before i released kV 😅️ but they had less motions than kV. was hoping they would keep pushing but seems they're dropping it or what?

Seems like they dropped a barebone implementation in Xcode 13 and then forgot about it. No ., and things are really broken when you do a / search while modifying the source between two n. They killed XVim in Xcode 14 and JetBrains announced a couple weeks ago that they stopped AppCode. So no good Vim solution for iOS development until I found kindaVim!

the problem is there's no macOS API to track a UI Element. you can track apps, you could track windows, but you can't track UI Elements. so it's not possible for kV to know when you are back to a previous element.

That sucks. For the sake of argument (as you seem open to it!), would it be possible to use a heuristic to identify a previous UI element? For example the combination of:

• The window ID (or AXUIElementGetPid?)
• The last known kAXValueAttribute before loosing focus
• The last known kAXSelectedTextRangeAttribute to disambiguate

This wouldn't work for a lot of UI elements, but for large text fields (such as Xcode's editor) I guess it could be helpful?

yes, so there's some issues with that too. the first one is that you can't detect through the AX API that you will leave a UI Element. you can be notified (like with the API that you're pointing below) ONCE the UI Element changed. again, because you can't track what was the previous one, you can't go back and discard the selection. [...] if you'd remove the block cursor/selection, then you'll not be able to apply anything on your kV Visual Mode selection.

I see. If the above heuristic is not possible to identify a previous UI element, how about having a special case when focusing an element having a single-character selection (the one emulating the Normal mode caret), to either trigger the Normal mode, or discard the single-character selection?

but would you always want that? i remember playing with this at the beginning, but i've found the experience completely inconsistent. what worked best, with my experience, is to go back to Insert Mode as soon as you leave the current UI Element

I barely scratched the surface of kindaVim, which I'm using mostly with Xcode at the moment. I guess when you use it to navigate through complex user interfaces then it makes sense to revert to Insert (which becomes really the default/normal mode in UI). Maybe choosing the default mode per app could be helpful? Or to have the default mode being Normal only for text fields?

[godbout]

Seems like they dropped a barebone implementation in Xcode 13 and then forgot about it. No ., and things are really broken when you do a / search while modifying the source between two n. They killed XVim in Xcode 14 and JetBrains announced a couple weeks ago that they stopped AppCode. So no good Vim solution for iOS development until I found kindaVim!

AppCode is dead??? 😱️

hmm then i don't get how come kV is not bringing me millions every month day.

For the sake of argument (as you seem open to it!)

definitely. haven't reach the bottom of it yet.

That sucks. For the sake of argument (as you seem open to it!), would it be possible to use a heuristic to identify a previous UI element? For example the combination of:

• The window ID (or AXUIElementGetPid?)
• The last known kAXValueAttribute before loosing focus
• The last known kAXSelectedTextRangeAttribute to disambiguate

This wouldn't work for a lot of UI elements, but for large text fields (such as Xcode's editor) I guess it could be helpful?

i've discovered after, working on an Alfred Workflow to bring windows to the foreground for Wooshy that there is a private API to grab the exact window ID with the AX APIs: _AXUIElementGetWindow

i guess there could be some heuristic yeah. maybe even window ID and UI Element position + size + role. could create a hash from those data, may work quite well. although if window is moves or UI Element resized (Text Area), that would coz issues. i think i thought about the value also, but it may be modified by other programs, so that would be an issue too. jfi, i'm definitely not a genius lol, and i'm not the best programmer by far, but i'm very clean, in the sense that 1) all that can be tested is tested (6k+ automated tests on kV) 2) every decision is not taken lightly, so there's (almost guaranteed) a reason why kV ends up the way it is. but still, we haven't reached the bottom of this yet i feel, so let's keep going.

I see. If the above heuristic is not possible to identify a previous UI element, how about having a special case when focusing an element having a single-character selection (the one emulating the Normal mode caret), to either trigger the Normal mode, or discard the single-character selection?

but again, would you always want that? you click on an Input Field, if there's only one character selected, you always want kV to enter Normal Mode? you tab through the Input, same? some other apps like Wooshy sends you there, same? you close a window and macOS reverts to focus on another Input, same? i'm still not comfortable with the possible inconsistency.

I barely scratched the surface of kindaVim, which I'm using mostly with Xcode at the moment. I guess when you use it to navigate through complex user interfaces then it makes sense to revert to Insert (which becomes really the default/normal mode in UI).

yeah personally i use it in Xcode, GitHub comments, iA Writer, etc. but also A LOT in UI. like choosing Alfred's results, going through menus etc. but we could separate Text from Non Text Elements (although not sure it's easy with the current structure of the Engines).

Maybe choosing the default mode per app could be helpful?

wouldn't work on a per app basis. an app will have different UI Elements. would need to be automatic, from the Element role. i think.

Or to have the default mode being Normal only for text fields?

so, is the summary like: go directly to Normal Mode when you enter a UI Element that is a Text one? 😅️😂️

[mickael-menu]

AppCode is dead??? 😱️

Sadly yeah 😞. You might get a bunch of new subscribers once Apple releases a new Xcode version breaking the compatibility with AppCode!

i guess there could be some heuristic yeah. maybe even window ID and UI Element position + size + role. could create a hash from those data, may work quite well. although if window is moves or UI Element resized (Text Area), that would coz issues. i think i thought about the value also, but it may be modified by other programs, so that would be an issue too.

IMHO that's worth testing with this hashing method. Reverting to Insert mode because the window was resized or the text field value changed separately from kindaVim sounds like a fine default to me as we refocus an element that is effectively in a different state than where it was left with kindaVim.

I think that would be the cleanest and more consistent way to handle this, compared to the single-character selection hack.

so, is the summary like: go directly to Normal Mode when you enter a UI Element that is a Text one?

Well that's how Vim works and I think I would actually prefer that (maybe an opt-in option?). More power in my text fields 😉

But maybe I'll get used to the way kindaVim works. I'm going back and forth between Xcode, Android Studio (with IDEAVim) and Neovim so right now it's a brainfuck every time I switch.

[godbout]

Sadly yeah 😞. You might get a bunch of new subscribers once Apple releases a new Xcode version breaking the compatibility with AppCode!

seems it's gonna hold until Dec 2023. plenty of time for me to die first ☹️

IMHO that's worth testing with this hashing method. Reverting to Insert mode because the window was resized or the text field value changed separately from kindaVim sounds like a fine default to me as we refocus an element that is effectively in a different state than where it was left with kindaVim.

I think that would be the cleanest and more consistent way to handle this, compared to the single-character selection hack.

alright then.

there's probably gonna be some issues when kV can't detect whether a UI Element is Text or Non Text. that does happen, and some apps even need to be put in the 911 Family for that reason. i guess it's still worth a try.

Well that's how Vim works and I think I would actually prefer that (maybe an opt-in option?). More power in my text fields 😉

true. although yeah, Vim owns its own thing (buffer and all). kV doesn't, which is where it gets hard. especially dealing with when kV is in Insert Mode—normal macOS—because there's many things that can happen and are out of kV's control.

But maybe I'll get used to the way kindaVim works. I'm going back and forth between Xcode, Android Studio (with IDEAVim) and Neovim so right now it's a brainfuck every time I switch.

well i'll research on my side, but it'd be very useful if you keep me updated on your usage yeah. curious to know if this is a deal breaker or not. no one mentioned about that before. and where adding motions is pretty straightforward (except for Le Dot and handling Insert Mode), making modifications on the Engines that deal with the whole macOS system is rather tricky. (usually quite a lot of side-effects, and mostly hard to get those things automatedly tested.)

[mickael-menu]

Thank you for considering it, I'm looking forward to it!

I'll keep you posted how it goes using kindaVim. I guess I need to get used to hitting Caps-Lock every time I focus a text field.

[mickael-menu]

I might have stumbled onto something useful. I was doing some tests and noticed that the AXUIElement.hashValue uniquely identify the element even after switching the focus.

Screen.Recording.2023-01-08.at.20.32.13.mov

[godbout]

oh noice. thank you! do you want to have access to kV's source? 😂️

but actually does it matter now that the Focused Element is the same, if we enter automatically into Normal Mode once you reach an Element that is of Text role?

tracking which Element is which would be useful if kV was tracking all the data needed to do its work EVEN when leaving an Element. which is definitely not the case. as soon as you leave the whole world is reset.

[godbout]

actually the .hashValue is just so right. i think i wasn't aware of it when i started kV (no knowledge of Mac dev and Swift). you can override your own if needed (i think) but the default may work just great. there's some cases, i don't remember in which app (kV? Ws? Sl?) where i think i make changes on how the hash value is generated and how things get compared. but can't remember where and why. still haven't coffeed here.

[mickael-menu]

but actually does it matter now that the Focused Element is the same, if we enter automatically into Normal Mode once you reach an Element that is of Text role?

I guess not, but if you want to keep close to Vim's behavior the mode should be persisted per element. That would make it more cohesive with other types of elements too. But I think that's not a deal breaker at all.

actually the .hashValue is just so right. i think i wasn't aware of it when i started kV

I actually asked for help to ChatGPT. 😁 Most of it was garbage but the hash was a good idea, so I tested it.

thank you! do you want to have access to kV's source?

I would love to, but full disclosure: I'm working on something like SketchyVim but written in Swift (most likely open source eventually). Since kindaVim is proprietary, I understand if you're uncomfortable sharing the secret sauce. I'm all for discussing techniques and ideas though. Eventually I'd like to use real Vim in Xcode and kindaVim for other UIs.

[godbout]

I guess not, but if you want to keep close to Vim's behavior the mode should be persisted per element. That would make it more cohesive with other types of elements too. But I think that's not a deal breaker at all.

but the analogy would be like running lots of Vim all over macOS 😅️ i mean, in terms of kV, is that really useful to keep track of each Element's world? that'd be way more CPU and RAM intensive. not sure it's worth it at the end? how often/critical is this needed?

I actually asked for help to ChatGPT. 😁 Most of it was garbage but the hash was a good idea, so I tested it.

wow LOL

I would love to,

i was making a joke 😂️ but why not. there's already some people on it. also kV was open-source for a while, but had the License for people who didn't want to bother. closed it because people couldn't behave. anyways ultimately i'm not sure you'll be able to "understand" the code. not that i'm judging your intelligence 😅️, not at all, but it's spread over multiple packages, the code is very personal, and mostly a reaction to my understanding of Vim and of the macOS APIs, and the complexity of trying to recreate Vim with those APIs. lots of the code ends up being a solution of a problem that i couldn't foresee, because i lacked knowledge or on Vim, or on the macOS APIs, or of a bug in those. but if you're very familiar with Vim and with the macOS APIs—especially the AX ones—then you may actually understand quite a big chunk, who knows. lemme digest the idea.

but full disclosure: I'm working on something like SketchyVim but written in Swift (most likely open source eventually). Since kindaVim is proprietary, I understand if you're uncomfortable sharing the secret sauce. I'm all for discussing techniques and ideas though. Eventually I'd like to use real Vim in Xcode and kindaVim for other UIs.

but mostly the approach is gonna be VERY different, no? kV started as SketchyVim, with a Vim backend. but i've found it was way more painful to try to make the bridging work, even more when it comes to moves that deal with Screen Lines. when it's text, it's fine (more or less, as long as the text is within the visible area). when it's about the screen (moves like H, zt) i think you need to use vanilla macOS APIs to handle those. edit: also in my case i wanted to deal with the UI too, and text inputs where the text was not readable. hence the path kV took. and more: once a move is done, it's done. the point is just to keep adding more and more. each new move is a new brick added, and that is here to stay. that works well with my way of doing things.

[godbout]

@mickael-menu ok i've invited you to a couple of repos. probably the ones where you may find the most info regarding the AX, etc. see if you find something useful. at the end hopefully the people who will end up using your app wouldn't be kV's customers anyways. (thoughts and prayers.) if you find stuff that could be done better or have some other ideas you can share for sure but big chance i'll not understand and will have to stick with my own code sorry 😂️

[mickael-menu]

but the analogy would be like running lots of Vim all over macOS 😅️ i mean, in terms of kV, is that really useful to keep track of each Element's world? that'd be way more CPU and RAM intensive. not sure it's worth it at the end? how often/critical is this needed?

Yeah probably not worth it in the context of kindaVim where we switch between a lot of elements.

also kV was open-source for a while, but had the License for people who didn't want to bother. closed it because people couldn't behave.

I feel you, I have a couple open source projects and it can be overwhelming at times 😄 Thank you for sharing!

but mostly the approach is gonna be VERY different, no? kV started as SketchyVim, with a Vim backend. but i've found it was way more painful to try to make the bridging work

Oh yeah, I might loose the few remaining hair I have with this! I really miss full Vim power with Xcode (., stuff like :sort, etc.), so I'll be focused on supporting it in multiline textfields (and only accessibility ones, as that's enough for Xcode). But for the rest of the user interface and also single line text fields, your approach with kindaVim makes more sense.

[godbout]

Oh yeah, I might loose the few remaining hair I have with this! I really miss full Vim power with Xcode (., stuff like :sort, etc.), so I'll be focused on supporting it in multiline textfields (and only accessibility ones, as that's enough for Xcode). But for the rest of the user interface and also single line text fields, your approach with kindaVim makes more sense.

SketchyVim doesn't have this? with the Vim backend it should work no? only issue might be with the visible area. i think SV works super well but for small bouts of text. as soon as you're out of the visible area it's a little more tricky.

regarding your original question, i'm having more thoughts. for example, i personally use the passthroughs a lot. like i'm in Xcode in Normal Mode, then i call Alfred with command space and start typing my requests. if we enter Normal Mode each time we enter a new Element, then i'd enter Alfred in Normal Mode and would have to explicitly switch to Insert Mode in order to be able to type. i can foresee some other cases where you wouldn't want to start in Normal Mode when you enter a field. so i'm wondering, is your feeling about this because you mostly switch back to Xcode? can you foresee if you'd have the same issues i mentioned above in your own use cases? i guess we could make this more fine-tuned/complex, like only go back to Normal Mode automatically if it's a TextArea only? but here we start again with some heavy inconsistencies (to my taste) in how kV would work. rather than always go into Normal Mode with escape or your jk or your hotkey, you'll have also to compute yourself whether you're in a TextArea or not. and also if kV can't detect if you're in a TextArea then it will never enter Normal Mode. still feels to me like many inconsistencies.

or adding a new Setting where you could drop apps where you want kV to enter Normal Mode automatically?

[mickael-menu]

SketchyVim doesn't have this? with the Vim backend it should work no? only issue might be with the visible area. i think SV works super well but for small bouts of text. as soon as you're out of the visible area it's a little more tricky.

SketchyVim updates the whole kAXValueAttribute every time the buffer changes, which loses the scroll offset. In practice it's not very usable for me. From my tests an approach where you modify a selected range of text might fix the issue. kindaVim doesn't have this issue in Xcode so it should be doable.

xcode.mov

regarding your original question, i'm having more thoughts.

You're right, I can see how that could be an issue... I'm not sure we can be consistent without saving the mode for the element. How about:

1. Monitor opened windows.
2. When the mode of a focus element changes, save it with the element hash in the context of its window.
3. When focusing an element, check if it has an already saved mode for the window.
4. When closing a window, discard all the modes associated with it.