Helix crashes when editing files with Russian text

[Linuxoid85]

Summary

Helix crashes when I edit a Markdown document that is part or all Cyrillic characters. An error is displayed in the terminal:

% hx README.md 
thread 'main' panicked at 'byte index 421 is not a char boundary; it is inside 'З' (bytes 420..422) of `тся.
  2. Когда началось её исполнение.
  3. Во сколько её исполнение закончилось.
  4. Какой код завершения она вернула.
2. Постепенное исполнение`[...]', helix-core/src/syntax.rs:1246:25
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

• System: ArchLinux
• Terminal: GNOME Console
• LC_ALL: en_US.UTF-8, then set ru_RU.UTF-8

Reproduction Steps

I tried this:

1. hx or RUST_BACKTRACE=1 hx -vv README.md
2. File with Cyrillic symbols.
3. Crash!

I expected this to happen:

Instead, this happened:

% RUST_BACKTRACE=1 hx -vv README.md 
thread 'main' panicked at 'byte index 421 is not a char boundary; it is inside 'З' (bytes 420..422) of `тся.
  4. Когда началось её исполнение.
  5. Во сколько её исполнение закончилось.
  6. Какой код завершения она вернула.
2. Постепенное исполнение`[...]', helix-core/src/syntax.rs:1246:25
stack backtrace:
   0: rust_begin_unwind
             at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:579:5
   1: core::panicking::panic_fmt
             at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8d
[helix.log](https://github.com/helix-editor/helix/files/11675725/helix.log)
bc/library/core/src/panicking.rs:64:14
   2: core::str::slice_error_fail_rt
   3: core::str::slice_error_fail
             at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/core/src/str/mod.rs:86:9
   4: core::ops::function::impls::<impl core::ops::function::FnMut<A> for &mut F>::call_mut
   5: tree_sitter::Parser::parse_with::read
   6: ts_lexer_start
   7: ts_parser__lex
   8: ts_parser_parse
   9: helix_core::syntax::LanguageLayer::parse
  10: std::thread::local::LocalKey<T>::with
  11: helix_core::syntax::Syntax::update
  12: helix_view::document::Document::apply_impl
  13: helix_view::document::Document::apply_inner
  14: helix_term::commands::insert::insert_char
  15: helix_term::ui::editor::EditorView::insert_mode
  16: <helix_term::ui::editor::EditorView as helix_term::compositor::Component>::handle_event
  17: helix_term::compositor::Compositor::handle_event
  18: tokio::runtime::park::CachedParkThread::block_on
  19: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
  20: hx::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Helix log

~/.cache/helix/helix.log

2023-06-07T01:14:52.142 helix_view::editor [ERROR] Failed to initialize the LSP for `source.bash` { cannot find binary path }
2023-06-07T01:16:16.213 helix_view::editor [ERROR] editor error: Async job failed: request 2 timed out
2023-06-07T01:16:16.947 helix_view::editor [ERROR] editor error: Async job failed: request 3 timed out
2023-06-07T01:16:17.890 helix_view::editor [ERROR] editor error: Async job failed: request 4 timed out
2023-06-07T01:16:18.480 helix_view::editor [ERROR] editor error: Async job failed: request 5 timed out
2023-06-07T01:16:19.122 helix_view::editor [ERROR] editor error: Async job failed: request 6 timed out
2023-06-07T01:16:20.050 helix_view::editor [ERROR] editor error: Async job failed: request 7 timed out
2023-06-07T01:16:20.768 helix_lsp::transport [ERROR] <- ServerError(-32801): content modified
2023-06-07T01:16:20.768 helix_lsp::transport [ERROR] Tried sending response into a closed channel (id=Num(2)), original request likely timed out
2023-06-07T01:16:20.768 helix_lsp::transport [ERROR] <- ServerError(-32801): content modified
2023-06-07T01:16:20.768 helix_lsp::transport [ERROR] Tried sending response into a closed channel (id=Num(3)), original request likely timed out
2023-06-07T01:16:20.768 helix_lsp::transport [ERROR] <- ServerError(-32801): content modified
2023-06-07T01:16:20.768 helix_lsp::transport [ERROR] Tried sending response into a closed channel (id=Num(4)), original request likely timed out
2023-06-07T01:16:20.768 helix_lsp::transport [ERROR] <- ServerError(-32801): content modified
2023-06-07T01:16:20.768 helix_lsp::transport [ERROR] Tried sending response into a closed channel (id=Num(5)), original request likely timed out
2023-06-07T01:16:20.768 helix_lsp::transport [ERROR] <- ServerError(-32801): content modified
2023-06-07T01:16:20.768 helix_lsp::transport [ERROR] Tried sending response into a closed channel (id=Num(6)), original request likely timed out
2023-06-07T01:16:20.768 helix_lsp::transport [ERROR] <- ServerError(-32801): content modified
2023-06-07T01:16:20.768 helix_lsp::transport [ERROR] Tried sending response into a closed channel (id=Num(7)), original request likely timed out
2023-06-07T01:16:20.768 helix_lsp::transport [ERROR] <- ServerError(-32801): content modified
2023-06-07T01:16:20.768 helix_lsp::transport [ERROR] <- ServerError(-32801): content modified
2023-06-07T01:16:20.768 helix_lsp::transport [ERROR] <- ServerError(-32801): content modified
2023-06-07T01:16:20.768 helix_lsp::transport [ERROR] <- ServerError(-32801): content modified
2023-06-07T01:16:20.769 helix_lsp::transport [ERROR] <- ServerError(-32801): content modified
2023-06-07T01:16:20.769 helix_lsp::transport [ERROR] <- ServerError(-32801): content modified
2023-06-07T01:16:20.769 helix_lsp::transport [ERROR] <- ServerError(-32801): content modified
2023-06-07T01:16:20.769 helix_lsp::transport [ERROR] <- ServerError(-32801): content modified
2023-06-07T01:16:20.769 helix_lsp::transport [ERROR] <- ServerError(-32801): content modified
2023-06-07T01:16:20.772 helix_view::editor [ERROR] editor error: Async job failed: protocol error: ServerError(-32801): content modified
2023-06-07T01:16:20.777 helix_view::editor [ERROR] editor error: Async job failed: protocol error: ServerError(-32801): content modified
2023-06-07T01:16:20.780 helix_view::editor [ERROR] editor error: Async job failed: protocol error: ServerError(-32801): content modified
2023-06-07T01:16:20.783 helix_view::editor [ERROR] editor error: Async job failed: protocol error: ServerError(-32801): content modified
2023-06-07T01:16:20.787 helix_view::editor [ERROR] editor error: Async job failed: protocol error: ServerError(-32801): content modified
2023-06-07T01:16:20.790 helix_view::editor [ERROR] editor error: Async job failed: protocol error: ServerError(-32801): content modified
2023-06-07T01:16:20.793 helix_view::editor [ERROR] editor error: Async job failed: protocol error: ServerError(-32801): content modified
2023-06-07T01:16:20.795 helix_view::editor [ERROR] editor error: Async job failed: protocol error: ServerError(-32801): content modified
2023-06-07T01:16:20.798 helix_view::editor [ERROR] editor error: Async job failed: protocol error: ServerError(-32801): content modified
2023-06-07T01:17:08.426 helix_view::editor [ERROR] editor error: Async job failed: request 23 timed out
2023-06-07T01:17:11.179 helix_view::editor [ERROR] editor error: Async job failed: request 24 timed out
2023-06-07T01:17:12.599 helix_view::editor [ERROR] editor error: Async job failed: request 25 timed out
2023-06-07T01:17:18.150 helix_view::editor [ERROR] editor error: Async job failed: request 26 timed out
2023-06-07T01:17:18.574 helix_view::editor [ERROR] editor error: Async job failed: request 27 timed out
2023-06-07T01:17:25.342 helix_lsp::transport [ERROR] Tried sending response into a closed channel (id=Num(23)), original request likely timed out
2023-06-07T01:17:25.346 helix_lsp::transport [ERROR] Tried sending response into a closed channel (id=Num(24)), original request likely timed out
2023-06-07T01:17:25.351 helix_lsp::transport [ERROR] Tried sending response into a closed channel (id=Num(25)), original request likely timed out
2023-06-07T01:17:25.356 helix_lsp::transport [ERROR] Tried sending response into a closed channel (id=Num(26)), original request likely timed out
2023-06-07T01:17:25.361 helix_lsp::transport [ERROR] Tried sending response into a closed channel (id=Num(27)), original request likely timed out
2023-06-07T01:50:30.700 helix_view::editor [ERROR] Failed to initialize the LSP for `source.toml` { cannot find binary path }
2023-06-07T01:52:25.359 helix_view::editor [ERROR] Failed to initialize the LSP for `source.md` { cannot find binary path }
2023-06-07T02:01:04.547 helix_view::editor [ERROR] Failed to initialize the LSP for `source.md` { cannot find binary path }
2023-06-07T02:12:47.361 helix_view::editor [ERROR] editor error: file modified by an external process, use :w! to overwrite
2023-06-07T02:12:51.063 helix_view::editor [ERROR] editor error: file modified by an external process, use :w! to overwrite
2023-06-07T02:13:35.044 helix_view::editor [ERROR] editor error: file modified by an external process, use :w! to overwrite
2023-06-07T02:18:00.319 helix_view::editor [ERROR] editor error: file modified by an external process, use :w! to overwrite
2023-06-07T02:30:10.570 helix_view::editor [ERROR] editor error: file modified by an external process, use :w! to overwrite
2023-06-07T02:30:38.528 helix_view::editor [ERROR] editor error: no such command: 'W'
2023-06-07T02:34:38.134 helix_view::editor [ERROR] Failed to initialize the LSP for `source.md` { cannot find binary path }
2023-06-07T02:35:34.436 helix_view::editor [ERROR] Failed to initialize the LSP for `source.md` { cannot find binary path }
2023-06-07T02:37:52.456 helix_view::editor [ERROR] Failed to initialize the LSP for `source.md` { cannot find binary path }
2023-06-07T02:38:27.862 helix_view::editor [ERROR] Failed to initialize the LSP for `source.md` { cannot find binary path }
2023-06-07T02:39:12.887 helix_view::editor [ERROR] Failed to initialize the LSP for `source.md` { cannot find binary path }
2023-06-07T12:52:08.411 helix_view::editor [ERROR] Failed to initialize the LSP for `source.md` { cannot find binary path }
2023-06-07T12:54:27.109 helix_view::editor [ERROR] Failed to initialize the LSP for `source.md` { cannot find binary path }
2023-06-07T13:04:28.145 helix_view::clipboard::provider [DEBUG] No native clipboard provider found. Yanking by OSC 52 and pasting will be internal to Helix
2023-06-07T13:04:28.149 helix_tui::backend::crossterm [DEBUG] The keyboard enhancement protocol is not supported in this terminal (checked in 3.225575ms)
2023-06-07T13:04:28.149 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:28.547 helix_term::application [DEBUG] received editor event: IdleTimer
2023-06-07T13:04:29.652 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:29.933 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:30.051 helix_term::application [DEBUG] received editor event: IdleTimer
2023-06-07T13:04:30.051 helix_view::editor [ERROR] editor error: no such command: 'Q'
2023-06-07T13:04:30.052 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:30.821 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:31.015 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:31.222 helix_term::application [DEBUG] received editor event: IdleTimer
2023-06-07T13:04:31.605 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:31.988 helix_term::commands::typed [DEBUG] quitting...
2023-06-07T13:04:31.989 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:32.172 helix_term::commands::typed [DEBUG] quitting...
2023-06-07T13:04:32.172 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:32.172 helix_term::job [DEBUG] waiting on jobs...
2023-06-07T13:04:32.172 helix_term::job [DEBUG] waiting on jobs...
2023-06-07T13:04:36.549 helix_view::clipboard::provider [DEBUG] No native clipboard provider found. Yanking by OSC 52 and pasting will be internal to Helix
2023-06-07T13:04:36.736 helix_vcs [INFO] file is untracked
2023-06-07T13:04:36.736 helix_vcs [INFO] failed to open diff base for /home/admin/calmira/CABS/Ports/src/mkport/README.md
2023-06-07T13:04:36.737 helix_view::editor [ERROR] Failed to initialize the LSP for `source.md` { cannot find binary path }
2023-06-07T13:04:36.737 helix_view::editor [DEBUG] editor status: Loaded 1 file.
2023-06-07T13:04:36.748 helix_tui::backend::crossterm [DEBUG] The keyboard enhancement protocol is not supported in this terminal (checked in 10.557613ms)
2023-06-07T13:04:36.750 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:36.950 helix_term::application [DEBUG] received editor event: IdleTimer
2023-06-07T13:04:37.211 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:37.610 helix_term::application [DEBUG] received editor event: IdleTimer
2023-06-07T13:04:37.712 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:37.743 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:37.774 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:37.804 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:37.832 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:37.864 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:37.897 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:37.931 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:37.982 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:37.987 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:38.014 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:38.044 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:38.074 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:38.104 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:38.134 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:38.167 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:38.198 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:38.231 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:38.265 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:38.298 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:38.311 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:38.344 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:38.374 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:38.407 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:38.435 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:38.464 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:38.496 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:38.531 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:38.566 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:38.602 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:38.614 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:38.644 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:38.675 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:38.707 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:38.740 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:38.768 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:38.811 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:38.844 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:38.856 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:38.886 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:38.916 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:38.953 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:39.000 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:39.009 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:39.033 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:39.274 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:39.468 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:39.861 helix_term::application [DEBUG] received editor event: IdleTimer
2023-06-07T13:04:39.968 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:39.999 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:40.041 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:40.071 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:40.100 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:40.115 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:40.513 helix_term::application [DEBUG] received editor event: IdleTimer
2023-06-07T13:04:40.601 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:40.814 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:41.212 helix_term::application [DEBUG] received editor event: IdleTimer
2023-06-07T13:04:41.214 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:41.269 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:41.275 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:41.584 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:41.720 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:41.878 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:42.047 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:42.191 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:42.314 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:42.419 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:42.563 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:42.691 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:42.806 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:42.950 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:43.064 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:43.177 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:43.307 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:43.406 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:43.625 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:43.739 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:43.918 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:44.303 helix_term::application [DEBUG] received editor event: IdleTimer
2023-06-07T13:04:44.307 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:44.521 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:44.714 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:44.919 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:45.037 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:45.131 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:45.341 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:45.489 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:45.837 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:45.961 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:46.273 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-06-07T13:04:46.397 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0

### ###

Platform

Linux

Terminal Emulator

GNOME Console 44.0

Helix Version

helix 23.05 (7f5940b)

[pascalkuthe]

Can you provide an example file where helix crashes (ideally with reproducible steps that lerad to a crash) .

This was also potentially fixed by #7417 and if not is likely duplicate of #6645 so if a reproduction case is not provided I will close this as stale

[Linuxoid85]

Open any Markdown file with Cyrillic characters, edit it and after some time Helix crashes.
README.md

[the-mikedavis]

What edits are you making? Can you narrow down the reproduction steps? I can't reproduce with random changes

[pascalkuthe]

I was able to reprsuce this. The problem is that TS asks for the byte range of a node during incremental parsing. That byte range is somehow not aligned to char boundaries which causes a panic in rope.byte_slice. Ropey doesn't expose and API to directly iterate raw byte chunks so we need to wrap the ropey chunks iterator (using chunsk_at_byte which is the only ropey method which doesn't panic for non-char aligned bytes) ti be able to feed non-utf8 aligned chars to ropey.

This will also fix some other crashes (I think there are multiple open issues caused by this).

It's potentially/likely a grammar bug that some nodes are not aligned to char boundaries (but I am but sure). I think we should fix the osnic either way

[l4l]

I was able to get steps to segfault the parser: just keep on pasting абвгдежзийклмнопрстуфхцчшщъыьэюя in the beginning of the file until it got crashed. For some reason it only reproduces at last release at arch (i.e 23.10) but not at master (f992c3b).

Not really sure it's the same issue as the stacktrace differs:

#0  0x00007f97ac07a70c in last_block (s=0x55dd2dbc6f50) at /usr/src/debug/helix/helix-23.10/runtime/grammars/sources/markdown/tree-sitter-markdown/src/scanner.c:236
#1  scan (s=0x55dd2dbc6f50, lexer=0x55dd2da859a0, valid_symbols=0x7f97ac0c90bc <ts_external_scanner_states+188>) at /usr/src/debug/helix/helix-23.10/runtime/grammars/sources/markdown/tree-sitter-markdown/src/scanner.c:1347
#2  0x000055dd2bb72313 in ts_parser__lex (parse_state=41, version=<optimized out>, self=0x55dd2da859a0) at src/./parser.c:427
#3  ts_parser__advance (allow_node_reuse=<optimized out>, version=0, self=0x55dd2da859a0) at src/./parser.c:1441
#4  ts_parser_parse (self=0x55dd2da859a0, old_tree=<optimized out>, input=...) at src/./parser.c:1933
#5  0x000055dd2b0efd66 in tree_sitter::Parser::parse_with<&[u8], helix_core::syntax::{impl#16}::parse::{closure_env#2}> (self=0x7f97acf95cf0, callback=0x7ffdcd978d00, old_tree=...) at /build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tree-sitter-0.20.10/binding_rust/lib.rs:537
#6  helix_core::syntax::LanguageLayer::parse (self=0x55dd2d8e12d8, parser=0x7f97acf95cf0, source=...) at helix-core/src/syntax.rs:1403
#7  helix_core::syntax::{impl#13}::update::{closure#1} (ts_parser=<optimized out>) at helix-core/src/syntax.rs:1125
#8  std::thread::local::LocalKey<core::cell::RefCell<helix_core::syntax::TsParser>>::try_with<core::cell::RefCell<helix_core::syntax::TsParser>, helix_core::syntax::{impl#13}::update::{closure_env#1}, core::result::Result<(), helix_core::syntax::Error>> (f=..., self=<optimized out>)
    at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/std/src/thread/local.rs:270
#9  std::thread::local::LocalKey<core::cell::RefCell<helix_core::syntax::TsParser>>::with<core::cell::RefCell<helix_core::syntax::TsParser>, helix_core::syntax::{impl#13}::update::{closure_env#1}, core::result::Result<(), helix_core::syntax::Error>> (f=...)
    at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/std/src/thread/local.rs:246
#10 helix_core::syntax::Syntax::update (self=0x55dd2daae1a0, old_source=..., source=..., changeset=<optimized out>) at helix-core/src/syntax.rs:1094
#11 0x000055dd2b778684 in helix_view::document::Document::apply_impl (self=0x55dd2daae000, transaction=<optimized out>, view_id=..., emit_lsp_notification=true) at helix-view/src/document.rs:1203
#12 0x000055dd2b779b74 in helix_view::document::Document::apply_inner (self=0x55dd2daae000, transaction=0x7ffdcd9795a0, view_id=..., emit_lsp_notification=true) at helix-view/src/document.rs:1296
#13 0x000055dd2b65dcf0 in helix_view::document::Document::apply (self=0x55dd2daae000, transaction=0x7ffdcd9795a0, view_id=...) at helix-view/src/document.rs:1308
#14 helix_term::commands::paste_impl (values=..., doc=0x55dd2daae000, view=0x55dd2dabb0a0, action=<optimized out>, count=1, mode=helix_view::document::Mode::Normal) at helix-term/src/commands.rs:4076
#15 0x000055dd2b65ea6a in helix_term::commands::paste (editor=0x7ffdcd97b378, register=<optimized out>, pos=<optimized out>, count=1) at helix-term/src/commands.rs:4153

Payload in parse_with

[111, 99, 115, 47, 96, 32, 208, 184, 32, 208, 178, 209, 139, 208, 191, 208, 190, 208, 187, 208, 189, 208, 184, 209, 130, 208, 181, 58, 10, 10, 96, 96, 96, 98, 97, 115, 104, 10, 109, 100, 98, 111, 111, 107, 32, 115, 101, 114, 118, 101, 32, 45, 45, 111, 112, 101, 110, 10, 96, 96, 96, 10, 10, 62, 32, 42, 42, 208, 146, 208, 189, 208, 184, 208, 188, 208, 176, 208, 189, 208, 184, 208, 181, 33, 42, 42, 10, 62, 10, 62, 32, 208, 148, 208, 187, 209, 143, 32, 209, 141, 209, 130, 208, 190, 208, 179, 208, 190, 32, 208, 178, 208, 176, 208, 188, 32, 208, 189, 208, 181, 208, 190, 208, 177, 209, 133, 208, 190, 208, 180, 208, 184, 208, 188, 32, 109, 100, 66, 111, 111, 107, 58, 10, 10, 96, 96, 96, 98, 97, 115, 104, 10, 99, 97, 114, 103, 111, 32, 105, 110, 115, 116, 97, 108, 108, 32, 109, 100, 98, 111, 111, 107, 10, 96, 96, 96, 10, 10, 35, 35, 35, 32, 208, 148, 208, 187, 209, 143, 32, 209, 128, 208, 176, 208, 183, 209, 128, 208, 176, 208, 177, 208, 190, 209, 130, 209, 135, 208, 184, 208, 186, 208, 190, 208, 178, 10, 10, 96, 96, 96, 98, 97, 115, 104, 10, 99, 97, 114, 103, 111, 32, 100, 111, 99, 32, 45, 45, 110, 111, 45, 100, 101, 112, 115, 32, 45, 45, 111, 112, 101, 110, 10, 96, 96, 96, 10]

The failing function:

https://github.com/MDeiml/tree-sitter-markdown/blob/aaf76797aa8ecd9a5e78e0ec3681941de6c945ee/tree-sitter-markdown/src/scanner.c#L238-L240

with open_blocks simply empty: {size = 0, capacity = 0, items = 0x0}

UPD: sometimes it panics at ropey::byte_slice or similar

[l4l]

Ok, here even more detailed case that also reproduces on master and latest release:

1. Put cursor in the beginning of the file
2. Then paste абвгдежзийклмнопрстуфхцчшщъыьэюя
3. Add new line
4. Paste cyrillic line at (2) exactly four times

now it should panic with byte_slice(): Byte range does not align with char boundaries: range 2263..2457

stack trace

#6 0x0000556f0826041b in ropey::slice::RopeSlice::byte_slice> (self=0x7ffd64a350e8, byte_range=...) at /home/kitsu/.cargo/registry/src/g.yxqyang.asia-1ecc6299db9ec823/ropey-1.6.1/src/slice.rs:703 #7 0x0000556f081f0470 in helix_core::syntax::HighlightConfiguration::injection_pair (self=0x556f0929e020, query_match=0x7ffd64a35910, source=...) at helix-core/src/syntax.rs:1824 #8 0x0000556f081f0801 in helix_core::syntax::HighlightConfiguration::injection_for_match (self=0x556f0929e020, query=0x556f0929e0c0, query_match=0x7ffd64a35910, source=...) at helix-core/src/syntax.rs:1856 #9 0x0000556f082713a6 in helix_core::syntax::{impl#13}::update::{closure#1} (ts_parser=0x7fa78b5f8cb0) at helix-core/src/syntax.rs:1148 #10 0x0000556f081f6b54 in std::thread::local::LocalKey>::try_with, helix_core::syntax::{impl#13}::update::{closure_env#1}, core::result::Result<(), helix_core::syntax::Error>> (self=0x556f08cd52f8, f=...) at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/thread/local.rs:445 #11 0x0000556f081f5cbd in std::thread::local::LocalKey>::with, helix_core::syntax::{impl#13}::update::{closure_env#1}, core::result::Result<(), helix_core::syntax::Error>> (self=0x556f08cd52f8, f=...) at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/thread/local.rs:421 #12 0x0000556f081ecaa7 in helix_core::syntax::Syntax::update (self=0x556f0942cf40, old_source=..., source=..., changeset=0x7ffd64a37390) at helix-core/src/syntax.rs:1094 #13 0x0000556f07681e94 in helix_view::document::Document::apply_impl (self=0x556f0942ceb0, transaction=0x7ffd64a37390, view_id=..., emit_lsp_notification=true) at helix-view/src/document.rs:1203 #14 0x0000556f0768289c in helix_view::document::Document::apply_inner (self=0x556f0942ceb0, transaction=0x7ffd64a37390, view_id=..., emit_lsp_notification=true) at helix-view/src/document.rs:1296 #15 0x0000556f07682ab1 in helix_view::document::Document::apply (self=0x556f0942ceb0, transaction=0x7ffd64a37390, view_id=...) at helix-view/src/document.rs:1308 #16 0x0000556f06b28c89 in helix_term::commands::paste_impl (values=..., doc=0x556f0942ceb0, view=0x556f093a1c90, action=helix_term::commands::Paste::Cursor, count=1, mode=helix_view::document::Mode::Insert) at helix-term/src/commands.rs:4076 #17 0x0000556f06b290c7 in helix_term::commands::paste_bracketed_value (cx=0x7ffd64a37828, contents=...) at helix-term/src/commands.rs:4087

Note, it won't work if just paste the whole input at once.

Panics at:
https://github.com/helix-editor/helix/blob/master/helix-core/src/syntax.rs#L1824

capture.node at injection_pair is '[QueryCapture { node: {Node code_fence_content (60, 0) - (74, 0)}, index: 0 }, QueryCapture { node: {Node code_fence_content (60, 0) - (74, 0)}, index: 1 }]

[AndyJado]

I got the same panic message when editting, but at difference line of source. Not yet reproducable, since I've lost some unsaved text..

• panic message

thread 'main' panicked at 'byte index 693 is not a char boundary; it is inside '偶' (bytes 692..695) of `。

• line of source

哈哈`[...]', /Users/a/.cargo/registry/src/g.yxqyang.asia-1ecc
6299db9ec823/unicode-segmentation-1.10.1/src/grapheme.rs:553:22