Skip to content

Build backend: Add direct builds to the resolver and installer #9621

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Dec 4, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions crates/uv-dispatch/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ doctest = false
workspace = true

[dependencies]
uv-build-backend = { workspace = true }
uv-build-frontend = { workspace = true }
uv-cache = { workspace = true }
uv-client = { workspace = true }
Expand All @@ -30,9 +31,11 @@ uv-pypi-types = { workspace = true }
uv-python = { workspace = true }
uv-resolver = { workspace = true }
uv-types = { workspace = true }
uv-version = { workspace = true }

anyhow = { workspace = true }
futures = { workspace = true }
itertools = { workspace = true }
rustc-hash = { workspace = true }
tokio = { workspace = true }
tracing = { workspace = true }
65 changes: 63 additions & 2 deletions crates/uv-dispatch/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@ use anyhow::{anyhow, Context, Result};
use futures::FutureExt;
use itertools::Itertools;
use rustc_hash::FxHashMap;
use tracing::{debug, instrument};

use tracing::{debug, instrument, trace};
use uv_build_backend::check_direct_build;
use uv_build_frontend::{SourceBuild, SourceBuildContext};
use uv_cache::Cache;
use uv_client::RegistryClient;
Expand Down Expand Up @@ -397,6 +397,67 @@ impl<'a> BuildContext for BuildDispatch<'a> {
.await?;
Ok(builder)
}

async fn direct_build<'data>(
&'data self,
source: &'data Path,
subdirectory: Option<&'data Path>,
output_dir: &'data Path,
build_kind: BuildKind,
version_id: Option<String>,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be a Option<&str>?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. It would be kinda annoying, but doable. I don't feel strongly.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's more converting back and forth now, but it's also move consistent with the rest of the codebase.

) -> Result<Option<String>> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is being returned here? And should it be a PathBuf?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh okay, it's documented below on the trait.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This initially happened because PEP 517 says that the build backend returns the filename as last line, and we passed that around as a string.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More specifically, the location on disk may not be normalized the same as WheelFilename is, so for non-uv build backends we have to pass out a string and only parse later, i added comments.

// Direct builds are a preview feature with the uv build backend.
if self.preview.is_disabled() {
trace!("Preview is disabled, not checking for direct build");
return Ok(None);
}

let source_tree = if let Some(subdir) = subdirectory {
source.join(subdir)
} else {
source.to_path_buf()
};

// Only perform the direct build if the backend is uv in a compatible version.
let identifier = version_id.unwrap_or_else(|| source_tree.display().to_string());
if !check_direct_build(&source_tree, &identifier) {
trace!("Requirements for direct build not matched: {identifier}");
return Ok(None);
}

debug!("Performing direct build for {identifier}");

let output_dir = output_dir.to_path_buf();
let filename = tokio::task::spawn_blocking(move || -> Result<String> {
let filename = match build_kind {
BuildKind::Wheel => uv_build_backend::build_wheel(
&source_tree,
&output_dir,
None,
uv_version::version(),
)?
.to_string(),
BuildKind::Sdist => uv_build_backend::build_source_dist(
&source_tree,
&output_dir,
uv_version::version(),
)?
.to_string(),
BuildKind::Editable => uv_build_backend::build_editable(
&source_tree,
&output_dir,
None,
uv_version::version(),
)?
.to_string(),
};
Ok(filename)
})
.await??
.to_string();

Ok(Some(filename))
}
}

/// Shared state used during resolution and installation.
Expand Down
39 changes: 29 additions & 10 deletions crates/uv-distribution/src/source/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1803,27 +1803,46 @@ impl<'a, T: BuildContext> SourceDistributionBuilder<'a, T> {
fs::create_dir_all(&cache_shard)
.await
.map_err(Error::CacheWrite)?;
let disk_filename = self
// Try a direct build if that isn't disabled and the uv build backend is used.
let disk_filename = if let Some(name) = self
.build_context
.setup_build(
.direct_build(
source_root,
subdirectory,
source_root,
Some(source.to_string()),
source.as_dist(),
source_strategy,
temp_dir.path(),
if source.is_editable() {
BuildKind::Editable
} else {
BuildKind::Wheel
},
BuildOutput::Debug,
Some(source.to_string()),
)
.await
.map_err(Error::Build)?
.wheel(temp_dir.path())
.await
.map_err(Error::Build)?;
{
name
} else {
self.build_context
.setup_build(
source_root,
subdirectory,
source_root,
Some(source.to_string()),
source.as_dist(),
source_strategy,
if source.is_editable() {
BuildKind::Editable
} else {
BuildKind::Wheel
},
BuildOutput::Debug,
)
.await
.map_err(Error::Build)?
.wheel(temp_dir.path())
.await
.map_err(Error::Build)?
};

// Read the metadata from the wheel.
let filename = WheelFilename::from_str(&disk_filename)?;
Expand Down
15 changes: 15 additions & 0 deletions crates/uv-types/src/traits.rs
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,21 @@ pub trait BuildContext {
build_kind: BuildKind,
build_output: BuildOutput,
) -> impl Future<Output = Result<Self::SourceDistBuilder>> + 'a;

/// Build by calling directly into the uv build backend without PEP 517, if possible.
///
/// Checks if the source tree uses uv as build backend. If not, it returns `Ok(None)`, otherwise
/// it builds and returns the name of the built file.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does it return in the case of an editable?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At this stage, the editable is a wheel that is indistinguishable from a regular wheel, except it contains a .pth file instead of the actual sources (but we still unpack it the regular way) and we're not allowed to cache it.

///
/// `version_id` is for error reporting only.
fn direct_build<'a>(
&'a self,
source: &'a Path,
subdirectory: Option<&'a Path>,
output_dir: &'a Path,
build_kind: BuildKind,
version_id: Option<String>,
) -> impl Future<Output = Result<Option<String>>> + 'a;
}

/// A wrapper for `uv_build::SourceBuild` to avoid cyclical crate dependencies.
Expand Down
Loading