[red-knot] Implement basic LSP server

[dhruvmanila]

Summary

This PR adds basic LSP implementation for the Red Knot project.

This is basically a fork of the existing ruff_server crate into a red_knot_server crate. The following are the main differences:

1. The Session stores a map from workspace root to the corresponding Red Knot database (RootDatabase).
2. The database is initialized with the newly implemented LSPSystem (implementation of System trait)
3. The LSPSystem contains the server index corresponding to each workspace and an underlying OS system implementation. For certain methods, the system first checks if there's an open document in LSP system and returns the information from that. Otherwise, it falls back to the OS system to get that information. These methods are path_metadata, read_to_string and read_to_notebook
4. Add as_any_mut method for System

Why fork?

Forking allows us to experiment with the functionalities that are specific to Red Knot. The architecture is completely different and so the requirements for an LSP implementation are different as well. For example, Red Knot only supports a single workspace, so the LSP system needs to map the multi-workspace support to each Red Knot instance. In the end, the server code isn't too big, it will be easier to implement Red Knot specific functionality without worrying about existing server limitations and it shouldn't be difficult to port the existing server.

Review

Most of the server files hasn't been changed. I'm going to list down the files that have been changed along with highlight the specific part of the file that's changed from the existing server code.

Changed files:

• Red Knot CLI implementation: https://github.com/astral-sh/ruff/pull/12624/files#diff-579596339a29d3212a641232e674778c339b446de33b890c7fdad905b5eb50e1
• In https://github.com/astral-sh/ruff/pull/12624/files#diff-b9a9041a8a2bace014bf3687c3ef0512f25e0541f112fad6131b14242f408db6, server capabilities have been updated, dynamic capability registration is removed
• In https://github.com/astral-sh/ruff/pull/12624/files#diff-b9a9041a8a2bace014bf3687c3ef0512f25e0541f112fad6131b14242f408db6, the API for clear_diagnostics now take in a Url instead of DocumentQuery as the document version doesn't matter when clearing diagnostics after a document is closed
• did_close, did_close_notebook, did_open, did_open_notebook are updated to open / close file from the corresponding Red Knot workspace
• The diagnostic handler is updated to request diagnostics from Red Knot
• The [Session::new] method in https://github.com/astral-sh/ruff/pull/12624/files#diff-55c96201296200c1cab37c8b0407b6c733381374b94be7ae50563bfe95264e4d is updated to construct the Red Knot databases for each workspace. It also contains the index_mut and MutIndexGuard implementation
• And, LSPSystem implementation is in https://github.com/astral-sh/ruff/pull/12624/files#diff-4ed62bd359c43b0bf1a13f04349dcd954966934bb8d544de7813f974182b489e

Test Plan

First, configure VS Code to use the red_knot binary

1. Build the red_knot binary by cargo build
2. Update the VS Code extension to specify the path to this binary

{
	"ruff.path": ["/path/to/ruff/target/debug/red_knot"]
}

3. Restart VS Code

Now, open a file containing red-knot specific diagnostics, close the file and validate that diagnostics disappear.

[github-actions]

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Formatter (stable)

ℹ️ ecosystem check encountered format errors. (no format changes; 1 project error)

openai/openai-cookbook (error)

warning: Detected debug build without --no-cache.
error: Failed to parse examples/Chat_finetuning_data_prep.ipynb:6:18:25: Unparenthesized generator expression cannot be used here
error: Failed to parse examples/chatgpt/gpt_actions_library/gpt_action_confluence.ipynb:15:1:5: Simple statements must be separated by newlines or semicolons
error: Failed to parse examples/chatgpt/gpt_actions_library/gpt_action_gmail.ipynb:15:1:1: Expected an expression
error: Failed to parse examples/chatgpt/gpt_actions_library/gpt_action_jira.ipynb:15:1:1: Expected an expression
error: Failed to parse examples/chatgpt/gpt_actions_library/gpt_action_notion.ipynb:15:1:1: Expected an expression
error: Failed to parse examples/chatgpt/gpt_actions_library/gpt_action_sharepoint_doc.ipynb:28:1:5: Simple statements must be separated by newlines or semicolons
error: Failed to parse examples/chatgpt/gpt_actions_library/gpt_action_sharepoint_text.ipynb:28:1:5: Simple statements must be separated by newlines or semicolons
error: Failed to parse examples/chatgpt/gpt_actions_library/gpt_action_sql_database.ipynb:2:2:5: Simple statements must be separated by newlines or semicolons
error: Failed to parse examples/chatgpt/gpt_actions_library/gpt_middleware_azure_function.ipynb:37:1:13: Simple statements must be separated by newlines or semicolons

Formatter (preview)

ℹ️ ecosystem check encountered format errors. (no format changes; 1 project error)

openai/openai-cookbook (error)

ruff format --preview

warning: Detected debug build without --no-cache.
error: Failed to parse examples/Chat_finetuning_data_prep.ipynb:6:18:25: Unparenthesized generator expression cannot be used here
error: Failed to parse examples/chatgpt/gpt_actions_library/gpt_action_confluence.ipynb:15:1:5: Simple statements must be separated by newlines or semicolons
error: Failed to parse examples/chatgpt/gpt_actions_library/gpt_action_gmail.ipynb:15:1:1: Expected an expression
error: Failed to parse examples/chatgpt/gpt_actions_library/gpt_action_jira.ipynb:15:1:1: Expected an expression
error: Failed to parse examples/chatgpt/gpt_actions_library/gpt_action_notion.ipynb:15:1:1: Expected an expression
error: Failed to parse examples/chatgpt/gpt_actions_library/gpt_action_sharepoint_doc.ipynb:28:1:5: Simple statements must be separated by newlines or semicolons
error: Failed to parse examples/chatgpt/gpt_actions_library/gpt_action_sharepoint_text.ipynb:28:1:5: Simple statements must be separated by newlines or semicolons
error: Failed to parse examples/chatgpt/gpt_actions_library/gpt_action_sql_database.ipynb:2:2:5: Simple statements must be separated by newlines or semicolons
error: Failed to parse examples/chatgpt/gpt_actions_library/gpt_middleware_azure_function.ipynb:37:1:13: Simple statements must be separated by newlines or semicolons

[MichaReiser]

This overall looks great. We may not have to add system_mut after all. I added an inline comment for that.

The other two things that we have to figure out are but I would recommend removing for now:

• Settings
• API exposed to the LSP: Calling system_path_to_file can panic if any other thread calls Handle::get_mut. I think the LSP should instead use a facade with a limited set of queries that wrapps the queries in RootDatabase::with_db, similar to rust analyzer. Any queries not exposed on the facade should be considered internal.

[MichaReiser]

We may want to query the os file system to retrieve the permissions on disk.

[dhruvmanila]

I guess it depends on how are the permissions being utilized? I don't see any usages currently. I think one potential use-case from an editor perspective would be to re-check the file when the permission is changed but the content remains the same. Is that so?

[MichaReiser]

There's no immediate use case for it anymore now that we have shifted towards building a type checker. The only use we have for permissions are the permission-lints.

[dhruvmanila]

I'll add a basic support for non-workspace files in a follow-up PR.

[MichaReiser]

Awesome. This looks great!

If you can find the time, it would be great to have a follow up PR to store the file on DocumentController rather than looking it everytime we call check_file (or in close etc).

We should also add a TODO that the LSP shouldn't use any queries and instead should only use methods exposed on RootDatabase.