Rust: extract crate graph #18228

aibaars · 2024-12-06T11:29:41Z

This pull request changes the Rust extractor to emit information about the crate graph, and the public modules, types, traits and function signatures contained in each crate. This should give our analyses information about names and types provided by the dependencies of a project. The information in the crate graph is represented as AST nodes without macro calls; type paths should be fully qualified and methods/functions have no bodies.

This PR currently does not extract generic type arguments , where clauses, and associated types and aliases. These should be added in a follow-up pull request.

rust/ql/lib/codeql/rust/elements.qll

rust/ql/lib/codeql/rust/elements/EnumVariant.qll

rust/extractor/src/generated/top.rs

rust/ql/lib/codeql/rust/elements.qll

rust/ql/lib/codeql/rust/elements/ValueItem.qll

rust/ql/lib/codeql/rust/elements/VariantData.qll

rust/ql/lib/codeql/rust/elements.qll

rust/ql/lib/codeql/rust/elements/ImplItem.qll

rust/ql/lib/codeql/rust/elements/internal/generated/ImplItem.qll

rust/extractor/src/crate_graph.rs

rust/ql/lib/codeql/rust/elements/internal/generated/Crate.qll

+private import codeql.rust.elements.internal.generated.Raw
+import codeql.rust.elements.Crate
+import codeql.rust.elements.internal.ElementImpl::Impl as ElementImpl
+import codeql.rust.elements.Module


rust/extractor/src/crate_graph.rs

…ion checks These elements depend on the version of the standard libraries and platform, and in addition no location information is extracted for them at the moment. f

hvitved · 2025-03-11T09:52:31Z

rust/ql/integration-tests/options/cfg/functions.ql

@@ -1,4 +1,5 @@
 import rust

 from Function f
+where exists(f.getLocation().getFile().getRelativePath())


How about adding

/** Holds if this element is from source code. */ predicate fromSource() { exists(this.getFile().getRelativePath()) }

to the class Locatable inside LocatableImpl.qll, and then replace all these calls with simply fromSource?

hvitved · 2025-03-11T09:53:12Z

rust/ql/lib/codeql/files/FileSystem.qll

@@ -55,7 +55,8 @@ class File extends Container, Impl::File {
        |
          node.getFile() = this and
          line = [/*loc.getStartLine(), */ loc.getEndLine()] and // ignore start locations for now as we're getting them wrong for things with a comment attached
-          not loc instanceof EmptyLocation
+          not loc instanceof EmptyLocation and
+          line > 0


Why can't line 0 contain code?

Lines start at 1. A location with a 0 line is pointing to a whole file .

hvitved · 2025-03-11T10:00:12Z

rust/ql/lib/codeql/rust/elements/internal/CrateImpl.qll

+    override string toString() {
+      result = "Crate(" + this.getName() + "@" + concat(this.getVersion()) + ")"
+    }


Suggested change

override string toString() {

result = "Crate(" + this.getName() + "@" + concat(this.getVersion()) + ")"

}

override string toString() {

result = strictconcat(int i | | this.toStringPart(i), " " order by i)

}

private string toStringPart(int i) {

i = 0 and result = "Crate("

or

i = 1 and result = this.getName()

or

i = 2 and result = "@"

or

i = 3 and result = this.getVersion()

or

i = 4 and result = ")"

}

hvitved · 2025-03-11T10:07:33Z

rust/ql/lib/codeql/rust/internal/AstConsistency.qll

@@ -24,7 +24,9 @@ query predicate multipleLocations(Locatable e) { strictcount(e.getLocation()) >
 /**
 * Holds if `e` does not have a `Location`.
 */
-query predicate noLocation(Locatable e) { not exists(e.getLocation()) }
+query predicate noLocation(Locatable e) {
+  not exists(e.getLocation()) and not e.(AstNode).getParentNode*() = any(Crate c).getModule()


Does this mean we only have crate information for entities that are not in the source code?

No, we have crate information for things in source code, but the crate information has no locations.

Just to double check: This is because for things in source code, we will now extract two entities, one as before, and a new one with crate information (for which we do not want to report missing locations)?

Yes, that's right. For items in source code, you'd have signatures without bodies and without locations.

hvitved · 2025-03-12T14:26:47Z

As discussed offline, re-exports like this are not currently extracted, so we should do that follow-up.

github-actions bot added the Rust Pull requests that update Rust code label Dec 6, 2024

github-advanced-security bot found potential problems Dec 6, 2024

View reviewed changes

aibaars force-pushed the aibaars/crate-graph branch 5 times, most recently from 48783f3 to e0ecf47 Compare December 9, 2024 16:56

github-advanced-security bot found potential problems Dec 9, 2024

View reviewed changes

rust/extractor/src/generated/top.rs Fixed Show fixed Hide fixed

rust/extractor/src/generated/top.rs Fixed Show fixed Hide fixed

rust/extractor/src/generated/top.rs Fixed Show fixed Hide fixed

github-advanced-security bot found potential problems Dec 9, 2024

View reviewed changes

rust/ql/lib/codeql/rust/elements.qll Fixed Show fixed Hide fixed

rust/ql/lib/codeql/rust/elements.qll Fixed Show fixed Hide fixed

rust/ql/lib/codeql/rust/elements/ValueItem.qll Fixed Show fixed Hide fixed

rust/ql/lib/codeql/rust/elements/VariantData.qll Fixed Show fixed Hide fixed

aibaars force-pushed the aibaars/crate-graph branch 5 times, most recently from 9f083b9 to 7419e77 Compare December 11, 2024 09:10

github-advanced-security bot found potential problems Dec 16, 2024

View reviewed changes

aibaars force-pushed the aibaars/crate-graph branch 3 times, most recently from 4c9e103 to 83111f3 Compare December 18, 2024 21:01

aibaars force-pushed the aibaars/crate-graph branch from 4782bc1 to b3ed09d Compare January 15, 2025 13:04

redsun82 reviewed Feb 12, 2025

View reviewed changes

rust/extractor/src/crate_graph.rs Outdated Show resolved Hide resolved

redsun82 reviewed Feb 12, 2025

View reviewed changes

rust/extractor/src/crate_graph.rs Show resolved Hide resolved

redsun82 reviewed Feb 12, 2025

View reviewed changes

rust/extractor/src/crate_graph.rs Outdated Show resolved Hide resolved

aibaars force-pushed the aibaars/crate-graph branch 2 times, most recently from c651fee to ed57708 Compare February 27, 2025 11:09

aibaars marked this pull request as ready for review February 27, 2025 11:47

aibaars requested a review from a team as a code owner February 27, 2025 11:47

github-advanced-security bot found potential problems Feb 27, 2025

View reviewed changes

aibaars force-pushed the aibaars/crate-graph branch 3 times, most recently from 95cbd9b to 7d08551 Compare February 27, 2025 13:21

aibaars dismissed redsun82’s stale review via 394bc66 March 4, 2025 11:27

redsun82 previously approved these changes Mar 4, 2025

View reviewed changes

rust/extractor/src/crate_graph.rs Outdated Show resolved Hide resolved

aibaars dismissed redsun82’s stale review via bbe51cb March 7, 2025 12:14

aibaars force-pushed the aibaars/crate-graph branch from 394bc66 to bbe51cb Compare March 7, 2025 12:14

aibaars added 15 commits March 10, 2025 10:16

Rust: add Crate to dbscheme

75ca04f

Rust: extract crate graph

e21a7f5

Rust: ignore crate graph elements from summary stats and AST no-locat…

98a4096

…ion checks These elements depend on the version of the standard libraries and platform, and in addition no location information is extracted for them at the moment. f

Rust: add ExtractionStep::CrateGraph

c601adf

Rust: QL Test: exclude crate graph related elements

9bba666

Rust: implement Crate::toString

8ec8824

Rust: add some tests for the crate graph

fce637f

Rust: update integration tests

e5b0397

Rust: improve modules test case

ff32119

Rust: crate_graph: extract only public items

8961a22

Rust: add file for root modules of crates

845f002

Run: misc/bazel/3rdparty/update_cargo_deps.sh

58d7933

Rust: fix crate graph after rust-analyzer upgrade

5855a12

Rust: update expected output

f17f375

Address review comment

9814aef

aibaars force-pushed the aibaars/crate-graph branch from bbe51cb to 9814aef Compare March 10, 2025 09:31

hvitved reviewed Mar 11, 2025

View reviewed changes

aibaars added 2 commits March 11, 2025 12:32

Rust: add Locatable::fromSource

66ab3a8

Rust: improve performance of Crate::toString

3991dc3

aibaars force-pushed the aibaars/crate-graph branch from 766a338 to 3991dc3 Compare March 11, 2025 11:57

hvitved approved these changes Mar 11, 2025

View reviewed changes

aibaars merged commit fa79dbc into main Mar 13, 2025
33 checks passed

aibaars deleted the aibaars/crate-graph branch March 13, 2025 09:00

hvitved mentioned this pull request Mar 13, 2025

Rust: Implement basic type inference in QL #18632

Merged

hvitved mentioned this pull request Apr 2, 2025

Rust: Take prelude into account when resolving paths #19157

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rust: extract crate graph #18228

Rust: extract crate graph #18228

aibaars commented Dec 6, 2024 •

edited

Loading

Check warning

hvitved Mar 11, 2025

hvitved Mar 11, 2025

aibaars Mar 11, 2025

hvitved Mar 11, 2025

hvitved Mar 11, 2025

aibaars Mar 11, 2025

hvitved Mar 11, 2025

aibaars Mar 11, 2025

hvitved commented Mar 12, 2025

-    override string toString() {
-      result = "Crate(" + this.getName() + "@" + concat(this.getVersion()) + ")"
-    }
+    override string toString() {
+      result = strictconcat(int i | | this.toStringPart(i), " " order by i)
+    }
+    private string toStringPart(int i) {
+      i = 0 and result = "Crate("
+      or
+      i = 1 and result = this.getName()
+      or
+      i = 2 and result = "@"
+      or
+      i = 3 and result = this.getVersion()
+      or
+      i = 4 and result = ")"
+    }

Rust: extract crate graph #18228

Rust: extract crate graph #18228

Conversation

aibaars commented Dec 6, 2024 • edited Loading

Check warning

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hvitved commented Mar 12, 2025

aibaars commented Dec 6, 2024 •

edited

Loading