Skip to content

Commit ffcf0fd

Browse files
authored
[6.3.0] Lockfile updates (#18894)
* Update lockfile function exception Add new exception for this function to handle any syntax errors or missing data within the lockfile Related: #18455 PiperOrigin-RevId: 538144339 Change-Id: I82160f3bff6598c26b3f99c824fe85ea86086c1f * Update lockfile writing logic to be event triggered. The writing will only happen at the end of resolution in the after command function (at this point, the module and all needed module extensions are resolved). Which solves the problem of reading and writing into the lockfile multiple times in one invocation. PiperOrigin-RevId: 540552139 Change-Id: I4a78412a388bde2ff7949d119831318c40d49047 # Conflicts: # src/main/java/com/google/devtools/build/lib/bazel/bzlmod/BazelDepGraphFunction.java # src/main/java/com/google/devtools/build/lib/bazel/bzlmod/BazelLockFileFunction.java # src/test/java/com/google/devtools/build/lib/bazel/bzlmod/BazelLockFileFunctionTest.java * Add module extension to lockfile PiperOrigin-RevId: 543445157 Change-Id: Ib32a2c9fdc22c5b228d78141fc9648b3ef5edf7d # Conflicts: # src/main/java/com/google/devtools/build/lib/bazel/bzlmod/GsonTypeAdapterUtil.java # src/main/java/com/google/devtools/build/lib/bazel/bzlmod/SingleExtensionEvalFunction.java * fixes * Fix tests * Enable lockfile by default and fix warning PiperOrigin-RevId: 545927412 Change-Id: I34e8531b8e396ccdfe0eecbee22cb68f76f969fc # Conflicts: # src/test/java/com/google/devtools/build/lib/analysis/RunfilesRepoMappingManifestTest.java # src/test/java/com/google/devtools/build/lib/bazel/bzlmod/BazelDepGraphFunctionTest.java # src/test/java/com/google/devtools/build/lib/bazel/bzlmod/BazelModuleResolutionFunctionTest.java # src/test/java/com/google/devtools/build/lib/query2/testutil/SkyframeQueryHelper.java # src/test/java/com/google/devtools/build/lib/rules/starlarkdocextract/StarlarkDocExtractTest.java * Add lockfile documentation PiperOrigin-RevId: 546085523 Change-Id: If287e6e143f8858d185f83a1630275520db65e44 # Conflicts: # site/en/_book.yaml * Fix null event issue If a change "only" occurred in a module extension, then the value of ModuleResolutionEvent is null -the skyvalue is cached and the event was never sent- then "combineModuleExtensions" function would crash with null pointer exception while trying to get the old usages. In this case we can just re-add the old module extensions from the lockfile because if the module resolution is the same, this means the usage didn't change. PiperOrigin-RevId: 546821911 Change-Id: Ie685cbab654d1c41403aebd31ddad91033be1d56 * Fix tests * Update lockfile function exception Add new exception for this function to handle any syntax errors or missing data within the lockfile Related: #18455 PiperOrigin-RevId: 538144339 Change-Id: I82160f3bff6598c26b3f99c824fe85ea86086c1f * Update lockfile writing logic to be event triggered. The writing will only happen at the end of resolution in the after command function (at this point, the module and all needed module extensions are resolved). Which solves the problem of reading and writing into the lockfile multiple times in one invocation. PiperOrigin-RevId: 540552139 Change-Id: I4a78412a388bde2ff7949d119831318c40d49047 # Conflicts: # src/main/java/com/google/devtools/build/lib/bazel/bzlmod/BazelDepGraphFunction.java # src/main/java/com/google/devtools/build/lib/bazel/bzlmod/BazelLockFileFunction.java # src/test/java/com/google/devtools/build/lib/bazel/bzlmod/BazelLockFileFunctionTest.java * Add module extension to lockfile PiperOrigin-RevId: 543445157 Change-Id: Ib32a2c9fdc22c5b228d78141fc9648b3ef5edf7d # Conflicts: # src/main/java/com/google/devtools/build/lib/bazel/bzlmod/GsonTypeAdapterUtil.java # src/main/java/com/google/devtools/build/lib/bazel/bzlmod/SingleExtensionEvalFunction.java * fixes * Fix tests * Enable lockfile by default and fix warning PiperOrigin-RevId: 545927412 Change-Id: I34e8531b8e396ccdfe0eecbee22cb68f76f969fc # Conflicts: # src/test/java/com/google/devtools/build/lib/analysis/RunfilesRepoMappingManifestTest.java # src/test/java/com/google/devtools/build/lib/bazel/bzlmod/BazelDepGraphFunctionTest.java # src/test/java/com/google/devtools/build/lib/bazel/bzlmod/BazelModuleResolutionFunctionTest.java # src/test/java/com/google/devtools/build/lib/query2/testutil/SkyframeQueryHelper.java # src/test/java/com/google/devtools/build/lib/rules/starlarkdocextract/StarlarkDocExtractTest.java * Add lockfile documentation PiperOrigin-RevId: 546085523 Change-Id: If287e6e143f8858d185f83a1630275520db65e44 # Conflicts: # site/en/_book.yaml * Fix null event issue If a change "only" occurred in a module extension, then the value of ModuleResolutionEvent is null -the skyvalue is cached and the event was never sent- then "combineModuleExtensions" function would crash with null pointer exception while trying to get the old usages. In this case we can just re-add the old module extensions from the lockfile because if the module resolution is the same, this means the usage didn't change. PiperOrigin-RevId: 546821911 Change-Id: Ie685cbab654d1c41403aebd31ddad91033be1d56 * Fix tests * Remove unrelated updates * reverse unrelated changes
1 parent d9658a9 commit ffcf0fd

37 files changed

+1316
-291
lines changed

site/en/_book.yaml

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -125,6 +125,21 @@ upper_tabs:
125125
path: /run/scripts
126126
- title: Client/server implementation
127127
path: /run/client-server
128+
- heading: External dependencies
129+
- title: Overview
130+
path: /external/overview
131+
- title: Bazel modules
132+
path: /external/module
133+
- title: Bazel registries
134+
path: /external/registry
135+
- title: Module extensions
136+
path: /external/extension
137+
- title: Module lockfile
138+
path: /external/lockfile
139+
- title: Bzlmod migration guide
140+
path: /external/migration
141+
- title: Advanced topics
142+
path: /external/advanced
128143
- heading: Querying your build
129144
- title: Query quickstart
130145
path: /query/quickstart

site/en/external/lockfile.md

Lines changed: 191 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,191 @@
1+
Project: /_project.yaml
2+
Book: /_book.yaml
3+
keywords: product:Bazel,lockfile,Bzlmod
4+
5+
# Bazel Lockfile
6+
7+
{% include "_buttons.html" %}
8+
9+
The lockfile feature in Bazel enables the recording of specific versions or
10+
dependencies of software libraries or packages required by a project. It
11+
achieves this by storing the result of module resolution and extension
12+
evaluation. The lockfile promotes reproducible builds, ensuring consistent
13+
development environments. Additionally, it enhances build efficiency by allowing
14+
Bazel to skip the resolution process when there are no changes in project
15+
dependencies. Furthermore, the lockfile improves stability by preventing
16+
unexpected updates or breaking changes in external libraries, thereby reducing
17+
the risk of introducing bugs.
18+
19+
## Lockfile Generation {:#lockfile-generation}
20+
21+
The lockfile is generated under the workspace root with the name
22+
`MODULE.bazel.lock`. It is created or updated during the build process,
23+
specifically after module resolution and extension evaluation. The lockfile
24+
captures the current state of the project, including the MODULE file, flags,
25+
overrides, and other relevant information. Importantly, it only includes
26+
dependencies that are included in the current invocation of the build.
27+
28+
When changes occur in the project that affect its dependencies, the lockfile is
29+
automatically updated to reflect the new state. This ensures that the lockfile
30+
remains focused on the specific set of dependencies required for the current
31+
build, providing an accurate representation of the project's resolved
32+
dependencies.
33+
34+
## Lockfile Usage {:#lockfile-usage}
35+
36+
The lockfile can be controlled by the flag
37+
[`--lockfile_mode`](/reference/command-line-reference#flag--lockfile_mode) to
38+
customize the behavior of Bazel when the project state differs from the
39+
lockfile. The available modes are:
40+
41+
* `update` (Default): If the project state matches the lockfile, the
42+
resolution result is immediately returned from the lockfile. Otherwise,
43+
resolution is executed, and the lockfile is updated to reflect the current
44+
state.
45+
* `error`: If the project state matches the lockfile, the resolution result is
46+
returned from the lockfile. Otherwise, Bazel throws an error indicating the
47+
variations between the project and the lockfile. This mode is particularly
48+
useful when you want to ensure that your project's dependencies remain
49+
unchanged, and any differences are treated as errors.
50+
* `off`: The lockfile is not checked at all.
51+
52+
## Lockfile Benefits {:#lockfile-benefits}
53+
54+
The lockfile offers several benefits and can be utilized in various ways:
55+
56+
- **Reproducible builds.** By capturing the specific versions or dependencies
57+
of software libraries, the lockfile ensures that builds are reproducible
58+
across different environments and over time. Developers can rely on
59+
consistent and predictable results when building their projects.
60+
61+
- **Efficient resolution skipping.** The lockfile enables Bazel to skip the
62+
resolution process if there are no changes in the project dependencies since
63+
the last build. This significantly improves build efficiency, especially in
64+
scenarios where resolution can be time-consuming.
65+
66+
- **Stability and risk reduction.** The lockfile helps maintain stability by
67+
preventing unexpected updates or breaking changes in external libraries. By
68+
locking the dependencies to specific versions, the risk of introducing bugs
69+
due to incompatible or untested updates is reduced.
70+
71+
## Lockfile Contents {:#lockfile-contents}
72+
73+
The lockfile contains all the necessary information to determine whether the
74+
project state has changed. It also includes the result of building the project
75+
in the current state. The lockfile consists of two main parts:
76+
77+
1. Inputs of the module resolution, such as `moduleFileHash`, `flags` and
78+
`localOverrideHashes`, as well as the output of the resolution, which is
79+
`moduleDepGraph`.
80+
2. For each module extension, the lockfile includes inputs that affect it,
81+
represented by `transitiveDigest`, and the output of running that extension
82+
referred to as `generatedRepoSpecs`
83+
84+
Here is an example that demonstrates the structure of the lockfile, along with
85+
explanations for each section:
86+
87+
```json
88+
{
89+
"lockFileVersion": 1,
90+
"moduleFileHash": "b0f47b98a67ee15f9.......8dff8721c66b721e370",
91+
"flags": {
92+
"cmdRegistries": [
93+
"https://bcr.bazel.build/"
94+
],
95+
"cmdModuleOverrides": {},
96+
"allowedYankedVersions": [],
97+
"envVarAllowedYankedVersions": "",
98+
"ignoreDevDependency": false,
99+
"directDependenciesMode": "WARNING",
100+
"compatibilityMode": "ERROR"
101+
},
102+
"localOverrideHashes": {
103+
"bazel_tools": "b5ae1fa37632140aff8.......15c6fe84a1231d6af9"
104+
},
105+
"moduleDepGraph": {
106+
"<root>": {
107+
"name": "",
108+
"version": "",
109+
"executionPlatformsToRegister": [],
110+
"toolchainsToRegister": [],
111+
"extensionUsages": [
112+
{
113+
"extensionBzlFile": "extension.bzl",
114+
"extensionName": "lockfile_ext"
115+
}
116+
],
117+
...
118+
}
119+
},
120+
"moduleExtensions": {
121+
"//:extension.bzl%lockfile_ext": {
122+
"transitiveDigest": "oWDzxG/aLnyY6Ubrfy....+Jp6maQvEPxn0pBM=",
123+
"generatedRepoSpecs": {
124+
"hello": {
125+
"bzlFile": "@@//:extension.bzl",
126+
...
127+
}
128+
}
129+
}
130+
}
131+
}
132+
```
133+
134+
### Module File Hash {:#module-file-hash}
135+
136+
The `moduleFileHash` represents the hash of the `MODULE.bazel` file contents. If
137+
any changes occur in this file, the hash value differs.
138+
139+
### Flags {:#flags}
140+
141+
The `Flags` object stores all the flags that can affect the resolution result.
142+
143+
### Local Override Hashes {:#local-override-hashes}
144+
145+
If the root module includes `local_path_overrides`, this section stores the hash
146+
of the `MODULE.bazel` file in the local repository. It allows tracking changes
147+
to this dependency.
148+
149+
### Module Dependency Graph {:#module-dep-graph}
150+
151+
The `moduleDepGraph` represents the result of the resolution process using the
152+
inputs mentioned above. It forms the dependency graph of all the modules
153+
required to run the project.
154+
155+
### Module Extensions {:#module-extensions}
156+
157+
The `moduleExtensions` section is a map that includes only the extensions used
158+
in the current invocation or previously invoked, while excluding any extensions
159+
that are no longer utilized. In other words, if an extension is not being used
160+
anymore across the dependency graph, it is removed from the `moduleExtensions`
161+
map.
162+
163+
Each entry in this map corresponds to a used extension and is identified by its
164+
containing file and name. The corresponding value for each entry contains the
165+
relevant information associated with that extension:
166+
167+
1. The `transitiveDigest` the digest of the extension implementation and its
168+
transitive .bzl files.
169+
2. The `generatedRepoSpecs` the result of running that extension with the
170+
current input.
171+
172+
An additional factor that can affect the extension results is their _usages_.
173+
Although not stored in the lockfile, the usages are considered when comparing
174+
the current state of the extension with the one in the lockfile.
175+
176+
## Best Practices {:#best-practices}
177+
178+
To maximize the benefits of the lockfile feature, consider the following best
179+
practices:
180+
181+
* Regularly update the lockfile to reflect changes in project dependencies or
182+
configuration. This ensures that subsequent builds are based on the most
183+
up-to-date and accurate set of dependencies.
184+
185+
* Include the lockfile in version control to facilitate collaboration and
186+
ensure that all team members have access to the same lockfile, promoting
187+
consistent development environments across the project.
188+
189+
By following these best practices, you can effectively utilize the lockfile
190+
feature in Bazel, leading to more efficient, reliable, and collaborative
191+
software development workflows.

src/main/java/com/google/devtools/build/lib/bazel/BUILD

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -142,6 +142,7 @@ java_library(
142142
"//src/main/java/com/google/devtools/build/lib:runtime",
143143
"//src/main/java/com/google/devtools/build/lib/analysis:blaze_version_info",
144144
"//src/main/java/com/google/devtools/build/lib/authandtls/credentialhelper:credential_module",
145+
"//src/main/java/com/google/devtools/build/lib/bazel/bzlmod:bazel_lockfile_module",
145146
"//src/main/java/com/google/devtools/build/lib/bazel/coverage",
146147
"//src/main/java/com/google/devtools/build/lib/bazel/debug:workspace-rule-module",
147148
"//src/main/java/com/google/devtools/build/lib/bazel/repository",

src/main/java/com/google/devtools/build/lib/bazel/Bazel.java

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,7 @@ public final class Bazel {
6363
com.google.devtools.build.lib.bazel.repository.RepositoryResolvedModule.class,
6464
com.google.devtools.build.lib.bazel.repository.CacheHitReportingModule.class,
6565
com.google.devtools.build.lib.bazel.SpawnLogModule.class,
66+
com.google.devtools.build.lib.bazel.bzlmod.BazelLockFileModule.class,
6667
com.google.devtools.build.lib.outputfilter.OutputFilteringModule.class,
6768
com.google.devtools.build.lib.worker.WorkerModule.class,
6869
com.google.devtools.build.lib.runtime.CacheFileDigestsModule.class,

src/main/java/com/google/devtools/build/lib/bazel/BazelRepositoryModule.java

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -151,7 +151,7 @@ public class BazelRepositoryModule extends BlazeModule {
151151
private final AtomicBoolean ignoreDevDeps = new AtomicBoolean(false);
152152
private CheckDirectDepsMode checkDirectDepsMode = CheckDirectDepsMode.WARNING;
153153
private BazelCompatibilityMode bazelCompatibilityMode = BazelCompatibilityMode.ERROR;
154-
private LockfileMode bazelLockfileMode = LockfileMode.OFF;
154+
private LockfileMode bazelLockfileMode = LockfileMode.UPDATE;
155155
private List<String> allowedYankedVersions = ImmutableList.of();
156156
private SingleExtensionEvalFunction singleExtensionEvalFunction;
157157

@@ -265,8 +265,7 @@ public ResolutionReason getResolutionReason() {
265265
.addSkyFunction(
266266
SkyFunctions.MODULE_FILE,
267267
new ModuleFileFunction(registryFactory, directories.getWorkspace(), builtinModules))
268-
.addSkyFunction(
269-
SkyFunctions.BAZEL_DEP_GRAPH, new BazelDepGraphFunction(directories.getWorkspace()))
268+
.addSkyFunction(SkyFunctions.BAZEL_DEP_GRAPH, new BazelDepGraphFunction())
270269
.addSkyFunction(
271270
SkyFunctions.BAZEL_LOCK_FILE, new BazelLockFileFunction(directories.getWorkspace()))
272271
.addSkyFunction(SkyFunctions.BAZEL_MODULE_INSPECTION, new BazelModuleInspectorFunction())

src/main/java/com/google/devtools/build/lib/bazel/bzlmod/BUILD

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,21 +85,43 @@ java_library(
8585
],
8686
)
8787

88+
java_library(
89+
name = "bazel_lockfile_module",
90+
srcs = ["BazelLockFileModule.java"],
91+
deps = [
92+
":resolution",
93+
":resolution_impl",
94+
"//src/main/java/com/google/devtools/build/lib:runtime",
95+
"//src/main/java/com/google/devtools/build/lib/bazel/bzlmod:module_extension",
96+
"//src/main/java/com/google/devtools/build/lib/bazel/repository:repository_options",
97+
"//src/main/java/com/google/devtools/build/lib/cmdline",
98+
"//src/main/java/com/google/devtools/build/lib/util:abrupt_exit_exception",
99+
"//src/main/java/com/google/devtools/build/lib/vfs",
100+
"//third_party:flogger",
101+
"//third_party:gson",
102+
"//third_party:guava",
103+
"//third_party:jsr305",
104+
],
105+
)
106+
88107
java_library(
89108
name = "resolution",
90109
srcs = [
91110
"AbridgedModule.java",
92111
"ArchiveOverride.java",
93112
"BazelDepGraphValue.java",
94113
"BazelLockFileValue.java",
114+
"BazelModuleResolutionEvent.java",
95115
"BazelModuleResolutionValue.java",
96116
"BzlmodFlagsAndEnvVars.java",
97117
"GitOverride.java",
98118
"InterimModule.java",
99119
"LocalPathOverride.java",
120+
"LockFileModuleExtension.java",
100121
"Module.java",
101122
"ModuleBase.java",
102123
"ModuleExtensionEvalStarlarkThreadContext.java",
124+
"ModuleExtensionResolutionEvent.java",
103125
"ModuleFileValue.java",
104126
"ModuleOverride.java",
105127
"MultipleVersionOverride.java",

0 commit comments

Comments
 (0)