Add support for remote_files for http_archive #22155

fzakaria · 2024-04-26T17:59:47Z

This issue adds support necessary to tackle
bazelbuild/bazel-central-registry#1566

Add two new attributes to http_archive: remote_file_urls and remote_file_integrity.

The purpose of these two attributes is to allow files to effectively be overlaid ontop of an http_archive. The goal of such functionality would be useful for BCR since the BUILD & WORKSPACE files need no longer be stored as patch files.

This means we could probably deprecate build_file since that could be referenced as a file:// url in the remote_file_urls attribute.

Co-authored-by: Mark Williams [email protected]

CC @fmeum

google-cla · 2024-04-26T17:59:52Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

tools/build_defs/repo/utils.bzl

src/test/shell/bazel/external_remote_file_test.sh

tools/build_defs/repo/http.bzl

This issue adds support necessary to tackle bazelbuild/bazel-central-registry#1566 Add two new attributes to http_archive: remote_file_urls and remote_file_integrity. The purpose of these two attributes is to allow files to effectively be overlaid ontop of an http_archive. The goal of such functionality would be useful for BCR since the BUILD & WORKSPACE files need no longer be stored as patch files. This means we could probably deprecate `build_file` since that could be referenced as a file:// url in the remote_file_urls attribute. Co-authored-by: Mark Williams <[email protected]>

* Added async download for remote_file for http-archive via block param * Added validation to check that there is no relative paths present

* Added disallow relative path * Added disallow absolute path * Added test for missing integrity

fmeum

LGTM with minor comments

tools/build_defs/repo/utils.bzl

src/test/shell/bazel/external_remote_file_test.sh

* Fixup comments * Add test that verifies remote_files works with more than 1 segment

src/test/shell/bazel/external_remote_file_test.sh

fmeum

@Wyverald Could you do the owner's review?

tools/build_defs/repo/http.bzl

Wyverald

very nice! just a few docs nits.

tools/build_defs/repo/http.bzl

tools/build_defs/repo/utils.bzl

fmeum · 2024-05-07T06:14:00Z

@bazel-io fork 7.2.0

@fmeum

This issue adds support necessary to tackle bazelbuild/bazel-central-registry#1566 Add two new attributes to http_archive: remote_file_urls and remote_file_integrity. The purpose of these two attributes is to allow files to effectively be overlaid ontop of an http_archive. The goal of such functionality would be useful for BCR since the BUILD & WORKSPACE files need no longer be stored as patch files. This means we could probably deprecate `build_file` since that could be referenced as a file:// url in the remote_file_urls attribute. Co-authored-by: Mark Williams <[email protected]> CC @fmeum Closes bazelbuild#22155. PiperOrigin-RevId: 632594203 Change-Id: I6310093482c5c58537ed6dbe4ff90bafdbd696ff

@fmeum

This issue adds support necessary to tackle bazelbuild/bazel-central-registry#1566 Add two new attributes to http_archive: remote_file_urls and remote_file_integrity. The purpose of these two attributes is to allow files to effectively be overlaid ontop of an http_archive. The goal of such functionality would be useful for BCR since the BUILD & WORKSPACE files need no longer be stored as patch files. This means we could probably deprecate `build_file` since that could be referenced as a file:// url in the remote_file_urls attribute. Co-authored-by: Mark Williams <[email protected]> CC @fmeum Closes #22155. PiperOrigin-RevId: 632594203 Change-Id: I6310093482c5c58537ed6dbe4ff90bafdbd696ff Commit e01509f Co-authored-by: Farid Zakaria <[email protected]>

@fmeum

This issue adds support necessary to tackle bazelbuild/bazel-central-registry#1566 Add two new attributes to http_archive: remote_file_urls and remote_file_integrity. The purpose of these two attributes is to allow files to effectively be overlaid ontop of an http_archive. The goal of such functionality would be useful for BCR since the BUILD & WORKSPACE files need no longer be stored as patch files. This means we could probably deprecate `build_file` since that could be referenced as a file:// url in the remote_file_urls attribute. Co-authored-by: Mark Williams <[email protected]> CC @fmeum Closes bazelbuild#22155. PiperOrigin-RevId: 632594203 Change-Id: I6310093482c5c58537ed6dbe4ff90bafdbd696ff

This is a continuation of bazelbuild#22155 that adds the newly added 'remote_files' attribute for http_archive to the bzlmod functionality. The end goal is to then update BCR to this new functionality to overlay files rather than use patch files when providing MODULE/WORKSPACE/BUILD files. bazelbuild/bazel-central-registry#1566 has a good discussion of the rationale.

iancha1992 · 2024-05-16T00:10:23Z

The changes in this PR have been included in Bazel 7.2.0 RC1. Please test out the release candidate and report any issues as soon as possible.
If you're using Bazelisk, you can point to the latest RC by setting USE_BAZEL_VERSION=7.2.0rc1. Thanks!

This is a continuation of #22155 that adds the newly added 'remote_files' attribute for http_archive to the bzlmod functionality. The end goal is to then update BCR to this new functionality to overlay files rather than use patch files when providing MODULE/WORKSPACE/BUILD files. bazelbuild/bazel-central-registry#1566 has a good discussion of the rationale. Co-authored-by: Fabian Meumertzheim <[email protected]> Closes #22349. PiperOrigin-RevId: 636682112 Change-Id: Ief070985598a7c0f427a98cd3daeb69a0984f7be

This is a continuation of bazelbuild#22155 that adds the newly added 'remote_files' attribute for http_archive to the bzlmod functionality. The end goal is to then update BCR to this new functionality to overlay files rather than use patch files when providing MODULE/WORKSPACE/BUILD files. bazelbuild/bazel-central-registry#1566 has a good discussion of the rationale. Co-authored-by: Fabian Meumertzheim <[email protected]> Closes bazelbuild#22349. PiperOrigin-RevId: 636682112 Change-Id: Ief070985598a7c0f427a98cd3daeb69a0984f7be

This is a continuation of #22155 that adds the newly added 'remote_files' attribute for http_archive to the bzlmod functionality. The end goal is to then update BCR to this new functionality to overlay files rather than use patch files when providing MODULE/WORKSPACE/BUILD files. bazelbuild/bazel-central-registry#1566 has a good discussion of the rationale. Co-authored-by: Fabian Meumertzheim <[email protected]> Closes #22349. PiperOrigin-RevId: 636682112 Change-Id: Ief070985598a7c0f427a98cd3daeb69a0984f7be Commit c4167e3 Co-authored-by: Farid Zakaria <[email protected]>

github-actions bot added the awaiting-review PR is awaiting review from an assigned reviewer label Apr 26, 2024

fzakaria commented Apr 26, 2024

View reviewed changes

tools/build_defs/repo/utils.bzl Show resolved Hide resolved

src/test/shell/bazel/external_remote_file_test.sh Outdated Show resolved Hide resolved

src/test/shell/bazel/external_remote_file_test.sh Show resolved Hide resolved

tools/build_defs/repo/http.bzl Show resolved Hide resolved

fzakaria mentioned this pull request Apr 26, 2024

[Idea] Support Starlark overlays without patches bazelbuild/bazel-central-registry#1566

Closed

fzakaria force-pushed the overlay-pr branch from 2899c6e to 840c498 Compare April 26, 2024 18:04

iancha1992 added the team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file. label Apr 26, 2024

fzakaria mentioned this pull request Apr 30, 2024

http.bzl : all_urls is given the remote_patch URL which is surely a mistake. #22201

Closed

fzakaria and others added 5 commits April 30, 2024 21:03

Update the MODULE.bazel.lock for test_repos

665f2ba

Remove spawn_strategy on account of bazelbuild#20753

7eab852

Added validation and async for remote_file

2e90536

* Added async download for remote_file for http-archive via block param * Added validation to check that there is no relative paths present

Updated lock files

a81db3c

fzakaria force-pushed the overlay-pr branch from d29d011 to a81db3c Compare April 30, 2024 21:09

fzakaria added 4 commits May 1, 2024 00:04

update MODULE.bazel.lock

4f150b9

Add back windows and macos python extension in MODULE.bazel.lock

8a0b673

Added more tests

d0d2970

* Added disallow relative path * Added disallow absolute path * Added test for missing integrity

Added test for bad integrity

1ebfbd1

fzakaria requested a review from fmeum May 1, 2024 18:06

fmeum approved these changes May 3, 2024

View reviewed changes

fmeum reviewed May 3, 2024

View reviewed changes

src/test/shell/bazel/external_remote_file_test.sh Outdated Show resolved Hide resolved

fmeum reviewed May 3, 2024

View reviewed changes

src/test/shell/bazel/external_remote_file_test.sh Outdated Show resolved Hide resolved

fmeum reviewed May 3, 2024

View reviewed changes

src/test/shell/bazel/external_remote_file_test.sh Show resolved Hide resolved

fzakaria added 3 commits May 3, 2024 16:35

Address feedback from fmeum

9f61f11

* Fixup comments * Add test that verifies remote_files works with more than 1 segment

Merge branch 'master' into overlay-pr

9b897bd

Update lock files

cf42f89

fzakaria requested a review from fmeum May 3, 2024 16:40

Add shard count to build rule

89e4878

fmeum reviewed May 3, 2024

View reviewed changes

src/test/shell/bazel/external_remote_file_test.sh Show resolved Hide resolved

fzakaria requested a review from fmeum May 3, 2024 18:25

fmeum approved these changes May 3, 2024

View reviewed changes

tools/build_defs/repo/http.bzl Outdated Show resolved Hide resolved

tools/build_defs/repo/http.bzl Outdated Show resolved Hide resolved

fzakaria added 3 commits May 6, 2024 17:30

Fixup docstring for remote_file_* attributes

6e558c0

Merge branch 'master' into overlay-pr

f330333

Update lock files

3cc59bb

fzakaria requested a review from fmeum May 6, 2024 17:40

update MODULE.bazel.lock

c91faf7

Wyverald approved these changes May 6, 2024

View reviewed changes

tools/build_defs/repo/http.bzl Outdated Show resolved Hide resolved

tools/build_defs/repo/http.bzl Outdated Show resolved Hide resolved

tools/build_defs/repo/utils.bzl Outdated Show resolved Hide resolved

Address feedback in PR for comments

e3ad0a9

fzakaria requested a review from Wyverald May 6, 2024 21:34

Wyverald approved these changes May 6, 2024

View reviewed changes

Wyverald added awaiting-PR-merge PR has been approved by a reviewer and is ready to be merge internally and removed awaiting-review PR is awaiting review from an assigned reviewer labels May 6, 2024

bazel-io mentioned this pull request May 7, 2024

[7.2.0] Add support for remote_files for http_archive #22271

Closed

copybara-service bot closed this in e01509f May 10, 2024

github-actions bot removed the awaiting-PR-merge PR has been approved by a reviewer and is ready to be merge internally label May 10, 2024

bazel-io mentioned this pull request May 10, 2024

[7.2.0] Add support for remote_files for http_archive #22325

Merged

fzakaria mentioned this pull request May 13, 2024

Add support for overlays in source.json #22349

Closed

bazel-io mentioned this pull request May 23, 2024

[7.2.0] Add support for overlays in source.json #22525

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add support for remote_files for http_archive #22155

Add support for remote_files for http_archive #22155

Uh oh!

fzakaria commented Apr 26, 2024 •

edited

Loading

Uh oh!

google-cla bot commented Apr 26, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fmeum left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fmeum left a comment

Uh oh!

Uh oh!

Uh oh!

Wyverald left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fmeum commented May 7, 2024

Uh oh!

iancha1992 commented May 16, 2024

Uh oh!

Uh oh!

Add support for remote_files for http_archive #22155

Add support for remote_files for http_archive #22155

Uh oh!

Conversation

fzakaria commented Apr 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

google-cla bot commented Apr 26, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fmeum left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fmeum left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Wyverald left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fmeum commented May 7, 2024

Uh oh!

iancha1992 commented May 16, 2024

Uh oh!

Uh oh!

fzakaria commented Apr 26, 2024 •

edited

Loading