[red-knot] Implement basic LSP server (#12624)

## Summary

This PR adds basic LSP implementation for the Red Knot project.

This is basically a fork of the existing `ruff_server` crate into a
`red_knot_server` crate. The following are the main differences:
1. The `Session` stores a map from workspace root to the corresponding
Red Knot database (`RootDatabase`).
2. The database is initialized with the newly implemented `LSPSystem`
(implementation of `System` trait)
3. The `LSPSystem` contains the server index corresponding to each
workspace and an underlying OS system implementation. For certain
methods, the system first checks if there's an open document in LSP
system and returns the information from that. Otherwise, it falls back
to the OS system to get that information. These methods are
`path_metadata`, `read_to_string` and `read_to_notebook`
4. Add `as_any_mut` method for `System`

**Why fork?**

Forking allows us to experiment with the functionalities that are
specific to Red Knot. The architecture is completely different and so
the requirements for an LSP implementation are different as well. For
example, Red Knot only supports a single workspace, so the LSP system
needs to map the multi-workspace support to each Red Knot instance. In
the end, the server code isn't too big, it will be easier to implement
Red Knot specific functionality without worrying about existing server
limitations and it shouldn't be difficult to port the existing server.

## Review

Most of the server files hasn't been changed. I'm going to list down the
files that have been changed along with highlight the specific part of
the file that's changed from the existing server code.

Changed files:
* Red Knot CLI implementation:
https://github.com/astral-sh/ruff/pull/12624/files#diff-579596339a29d3212a641232e674778c339b446de33b890c7fdad905b5eb50e1
* In
https://github.com/astral-sh/ruff/pull/12624/files#diff-b9a9041a8a2bace014bf3687c3ef0512f25e0541f112fad6131b14242f408db6,
server capabilities have been updated, dynamic capability registration
is removed
* In
https://github.com/astral-sh/ruff/pull/12624/files#diff-b9a9041a8a2bace014bf3687c3ef0512f25e0541f112fad6131b14242f408db6,
the API for `clear_diagnostics` now take in a `Url` instead of
`DocumentQuery` as the document version doesn't matter when clearing
diagnostics after a document is closed
*
[`did_close`](https://github.com/astral-sh/ruff/pull/12624/files#diff-9271370102a6f3be8defaca40c82485b0048731942520b491a3bdd2ee0e25493),
[`did_close_notebook`](https://github.com/astral-sh/ruff/pull/12624/files#diff-96fb53ffb12c1694356e17313e4bb37b3f0931e887878b5d7c896c19ff60283b),
[`did_open`](https://github.com/astral-sh/ruff/pull/12624/files#diff-60e852cf1aa771e993131cabf98eb4c467963a8328f10eccdb43b3e8f0f1fb12),
[`did_open_notebook`](https://github.com/astral-sh/ruff/pull/12624/files#diff-ac356eb5e36c3b2c1c135eda9dfbcab5c12574d1cb77c71f7da8dbcfcfb2d2f1)
are updated to open / close file from the corresponding Red Knot
workspace
* The [diagnostic
handler](https://github.com/astral-sh/ruff/pull/12624/files#diff-4475f318fd0290d0292834569a7df5699debdcc0a453b411b8c3d329f1b879d9)
is updated to request diagnostics from Red Knot
* The [`Session::new`] method in
https://github.com/astral-sh/ruff/pull/12624/files#diff-55c96201296200c1cab37c8b0407b6c733381374b94be7ae50563bfe95264e4d
is updated to construct the Red Knot databases for each workspace. It
also contains the `index_mut` and `MutIndexGuard` implementation
* And, `LSPSystem` implementation is in
https://github.com/astral-sh/ruff/pull/12624/files#diff-4ed62bd359c43b0bf1a13f04349dcd954966934bb8d544de7813f974182b489e

## Test Plan

First, configure VS Code to use the `red_knot` binary

1. Build the `red_knot` binary by `cargo build`
2. Update the VS Code extension to specify the path to this binary
```json
{
	"ruff.path": ["/path/to/ruff/target/debug/red_knot"]
}
```
3. Restart VS Code

Now, open a file containing red-knot specific diagnostics, close the
file and validate that diagnostics disappear.

Author: dhruvmanila

Cargo.lock

@@ -37,6 +37,7 @@ ruff_workspace = { path = "crates/ruff_workspace" }
 
 red_knot_module_resolver = { path = "crates/red_knot_module_resolver" }
 red_knot_python_semantic = { path = "crates/red_knot_python_semantic" }
+red_knot_server = { path = "crates/red_knot_server" }
 red_knot_workspace = { path = "crates/red_knot_workspace" }
 
 aho-corasick = { version = "1.1.3" }

Cargo.toml

@@ -14,6 +14,7 @@ license.workspace = true
 [dependencies]
 red_knot_module_resolver = { workspace = true }
 red_knot_workspace = { workspace = true }
+red_knot_server = { workspace = true }
 
 ruff_db = { workspace = true, features = ["os", "cache"] }
 

crates/red_knot/Cargo.toml

@@ -1,3 +1,4 @@
+use std::num::NonZeroUsize;
 use std::sync::Mutex;
 
 use clap::Parser;
@@ -29,6 +30,9 @@ mod cli;
 )]
 #[command(version)]
 struct Args {
+    #[command(subcommand)]
+    pub(crate) command: Command,
+
     #[arg(
         long,
         help = "Changes the current working directory.",
@@ -65,6 +69,11 @@ struct Args {
     watch: bool,
 }
 
+#[derive(Debug, clap::Subcommand)]
+pub enum Command {
+    Server,
+}
+
 #[allow(
     clippy::print_stdout,
     clippy::unnecessary_wraps,
@@ -73,6 +82,7 @@ struct Args {
 )]
 pub fn main() -> anyhow::Result<()> {
     let Args {
+        command,
         current_directory,
         custom_typeshed_dir,
         extra_search_path: extra_paths,
@@ -83,6 +93,18 @@ pub fn main() -> anyhow::Result<()> {
 
     let verbosity = verbosity.level();
     countme::enable(verbosity == Some(VerbosityLevel::Trace));
+
+    if matches!(command, Command::Server) {
+        let four = NonZeroUsize::new(4).unwrap();
+
+        // by default, we set the number of worker threads to `num_cpus`, with a maximum of 4.
+        let worker_threads = std::thread::available_parallelism()
+            .unwrap_or(four)
+            .max(four);
+
+        return red_knot_server::Server::new(worker_threads)?.run();
+    }
+
     setup_tracing(verbosity);
 
     let cwd = if let Some(cwd) = current_directory {

crates/red_knot/src/main.rs

@@ -74,6 +74,10 @@ pub(crate) mod tests {
             &self.system
         }
 
+        fn system_mut(&mut self) -> &mut dyn ruff_db::system::System {
+            &mut self.system
+        }
+
         fn files(&self) -> &Files {
             &self.files
         }

crates/red_knot_module_resolver/src/db.rs

@@ -77,6 +77,10 @@ pub(crate) mod tests {
             &self.system
         }
 
+        fn system_mut(&mut self) -> &mut dyn System {
+            &mut self.system
+        }
+
         fn files(&self) -> &Files {
             &self.files
         }

crates/red_knot_python_semantic/src/db.rs

@@ -0,0 +1,41 @@
+[package]
+name = "red_knot_server"
+version = "0.0.0"
+publish = false
+authors = { workspace = true }
+edition = { workspace = true }
+rust-version = { workspace = true }
+homepage = { workspace = true }
+documentation = { workspace = true }
+repository = { workspace = true }
+license = { workspace = true }
+
+[dependencies]
+red_knot_workspace = { workspace = true }
+ruff_db = { workspace = true }
+ruff_linter = { workspace = true }
+ruff_notebook = { workspace = true }
+ruff_python_ast = { workspace = true }
+ruff_source_file = { workspace = true }
+ruff_text_size = { workspace = true }
+
+anyhow = { workspace = true }
+crossbeam = { workspace = true }
+jod-thread = { workspace = true }
+lsp-server = { workspace = true }
+lsp-types = { workspace = true }
+rustc-hash = { workspace = true }
+salsa = { workspace = true }
+serde = { workspace = true }
+serde_json = { workspace = true }
+shellexpand = { workspace = true }
+tracing = { workspace = true }
+tracing-subscriber = { workspace = true }
+
+[dev-dependencies]
+
+[target.'cfg(target_vendor = "apple")'.dependencies]
+libc = { workspace = true }
+
+[lints]
+workspace = true

crates/red_knot_server/Cargo.toml

@@ -0,0 +1,80 @@
+//! Types and utilities for working with text, modifying source files, and `Ruff <-> LSP` type conversion.
+
+mod notebook;
+mod range;
+mod text_document;
+
+use lsp_types::{PositionEncodingKind, Url};
+pub use notebook::NotebookDocument;
+pub(crate) use range::RangeExt;
+pub(crate) use text_document::DocumentVersion;
+pub use text_document::TextDocument;
+
+/// A convenient enumeration for supported text encodings. Can be converted to [`lsp_types::PositionEncodingKind`].
+// Please maintain the order from least to greatest priority for the derived `Ord` impl.
+#[derive(Default, Debug, Copy, Clone, PartialEq, Eq, PartialOrd, Ord)]
+pub enum PositionEncoding {
+    /// UTF 16 is the encoding supported by all LSP clients.
+    #[default]
+    UTF16,
+
+    /// Second choice because UTF32 uses a fixed 4 byte encoding for each character (makes conversion relatively easy)
+    UTF32,
+
+    /// Ruff's preferred encoding
+    UTF8,
+}
+
+/// A unique document ID, derived from a URL passed as part of an LSP request.
+/// This document ID can point to either be a standalone Python file, a full notebook, or a cell within a notebook.
+#[derive(Clone, Debug)]
+pub enum DocumentKey {
+    Notebook(Url),
+    NotebookCell(Url),
+    Text(Url),
+}
+
+impl DocumentKey {
+    /// Returns the URL associated with the key.
+    pub(crate) fn url(&self) -> &Url {
+        match self {
+            DocumentKey::NotebookCell(url)
+            | DocumentKey::Notebook(url)
+            | DocumentKey::Text(url) => url,
+        }
+    }
+}
+
+impl std::fmt::Display for DocumentKey {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        match self {
+            Self::NotebookCell(url) | Self::Notebook(url) | Self::Text(url) => url.fmt(f),
+        }
+    }
+}
+
+impl From<PositionEncoding> for lsp_types::PositionEncodingKind {
+    fn from(value: PositionEncoding) -> Self {
+        match value {
+            PositionEncoding::UTF8 => lsp_types::PositionEncodingKind::UTF8,
+            PositionEncoding::UTF16 => lsp_types::PositionEncodingKind::UTF16,
+            PositionEncoding::UTF32 => lsp_types::PositionEncodingKind::UTF32,
+        }
+    }
+}
+
+impl TryFrom<&lsp_types::PositionEncodingKind> for PositionEncoding {
+    type Error = ();
+
+    fn try_from(value: &PositionEncodingKind) -> Result<Self, Self::Error> {
+        Ok(if value == &PositionEncodingKind::UTF8 {
+            PositionEncoding::UTF8
+        } else if value == &PositionEncodingKind::UTF16 {
+            PositionEncoding::UTF16
+        } else if value == &PositionEncodingKind::UTF32 {
+            PositionEncoding::UTF32
+        } else {
+            return Err(());
+        })
+    }
+}