feat(publish): Stabilize multi-package publishing #15636
+153
−855
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
A user will now be able to use flags like `--workspace` with `cargo publish`. `cargo package` will now also work with those flags without having to pass `--no-verify --exclude-lockfile`. Many release tools have come out that solve this problem. They will still need a lot of the logic that went into that for other parts of the release process. However, a cargo-native solution allows for: - Verification during dry-run - Better strategies for waiting for the publish timeout `cargo publish` is non-atomic at this time. If there is a server side error, network error, or rate limit during the publish, the workspace will be left in a partially published state. Verification is done before any publishing so that won't affect things. There are multiple strategies we can employ for improving this over time, including - atomic publish - `--idempotent` (rust-lang#13397) - leave this to release tools to manage This includes support for `--dry-run` verification. As release tools didn't have a way to do this before, users may be surprised at how slow this is because a `cargo build` is done instead of a `cargo check`. This is being tracked in rust-lang#14941. This adds to `cargo package` the `--registry` and `--index` flags to help with resolving dependencies when depending on a package being packaged at that moment. These flags are only needed when a `cargo package --workspace` operation would have failed before due to inability to find a locally created dependency. Regarding the publish timeout, `cargo publish --workspace` publishes packages in batches and we only timeout if nothing in the batch has finished being published within the timeout, deferring the rest to the next wait-for-publish. So for example, if you have packages `a`, `b`, `c` then we'll wait up to 60 seconds and if only `a` and `b` were ready in that time, we'll then wait another 60 seconds for `c`. During testing, users ran into issues with `.crate` checksums that we've not been able to reproduce since: - rust-lang#1169 (comment) - rust-lang#14396 By stabilizing this, Cargo's behavior becomes dependent on an overlay registry. When generating a lockfile or verifying a package, we overlay the locally generated `.crate` files on top of the registry so the registry appears as it would and everything works. If there is a conflict with a version, the local version wins which is important for the dry-run mode of release tools as they won't have bumped the version yet. Our concern for the overlay registry is dependency confusion attacks. Considering this is not accessible for general user operations, this should be fine. Fixes rust-lang#1169 Fixes rust-lang#10948
|
r? @weihanglo rustbot has assigned @weihanglo. Use |
rustbot
added
A-cli
|
@rfcbot fcp merge See the PR description for details |
|
Team member @epage has proposed to merge this. The next step is review by the rest of the tagged team members: No concerns currently listed. Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up! See this document for info about what commands tagged team members can give me. |
rfcbot
added
proposed-final-comment-period
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR try to resolve?
A user will now be able to use flags like
--workspacewithcargo publish.cargo packagewill now also work with those flags without having to pass--no-verify --exclude-lockfile.Many release tools have come out that solve this problem. They will still need a lot of the logic that went into that for other parts of the release process.
However, a cargo-native solution allows for:
cargo publishis non-atomic at this time.If there is a server side error, network error, or rate limit during the publish, the workspace will be left in a partially published state. Verification is done before any publishing so that won't affect things. There are multiple strategies we can employ for improving this over time, including
--idempotent(Want cargo publish --idempotent #13397)This includes support for
--dry-runverification. As release tools didn't have a way to do this before, users may be surprised at how slow this is because acargo buildis done instead of acargo check. This is being tracked in #14941.This adds to
cargo packagethe--registryand--indexflags to help with resolving dependencies when depending on a package being packaged at that moment.These flags are only needed when a
cargo package --workspaceoperation would have failed before due to inability to find a locally created dependency.Regarding the publish timeout,
cargo publish --workspacepublishes packages in batches and we only timeout if nothing in the batch has finished being published within the timeout, deferring the rest to the next wait-for-publish. So for example, if you have packagesa,b,cthen we'll wait up to 60 seconds and if onlyaandbwere ready in that time, we'll then wait another 60 seconds forc.During testing, users ran into issues with
.cratechecksums that we've not been able to reproduce since:-Zpackage-workspace#14396Fixes #1169
Fixes #10948
How to test and review this PR?
By stabilizing this, Cargo's behavior becomes dependent on an overlay registry.
When generating a lockfile or verifying a package, we overlay the locally generated
.cratefiles on top of the registry so the registry appears as it would and everything works.If there is a conflict with a version, the local version wins which is important for the dry-run mode of release tools as they won't have bumped the version yet.
Our concern for the overlay registry is dependency confusion attacks. Considering this is not accessible for general user operations, this should be fine.