Offline bootstrap

[zuiderkwast]

Possibility to build rebar3 itself in a build environment without Internet access.

Build environments may be forbidden to download any online content. For security and legal reasons, all dependencies may need to be downloaded and verified in advance. This is the case at some enterprises including Ericsson's internal projects.

Most of the online fetching is possible to work around (e.g. using _checkouts) but not bootstrapping.

Current behaviour

The bootstrap script fetches dependencies from Hex using inets httpc.

Related:

• rebar in offline mode? #1281
• https://stackoverflow.com/questions/68854474/is-it-possible-to-run-bootstrap-for-rebar3-offline

Expected behaviour

Some way to provide the dependencies to the bootstrap script, for example using _checkouts.

I'm willing to contribute a solution. Please let me know if you have a preferred solution in mind.

[ferd]

This is a bit tricky, but should be doable:

• the fetch code is currently able to see if a dep exists to skip it
• to ensure safe bootstrapping we generally remove everything in _build at first. This is because if there are files existing in there that change how the compiler works, then a previous build's artifacts can pollute the current bootstrap.

The easier attempt is to add a flag to bypass file clean-up. It won't generate builds that are as safe, so maybe just removing the ebin/ content could be decent enough in bypass mode?

[zuiderkwast]

Thx @ferd, I can try that.

It won't generate builds that are as safe

That's onfortunate since the idea is to make it more safe. :-) What if we can check the hash of the offline deps to match what's in the lock file?

In the long term, we would prefer a solution which is less of a hack. A genuine offline mode. Preferably, we'd also like to see decoupled fetching and building and have a command which can check if all dependencies are available before building them and report which dependencies are missing, rather than trying to download them.

[ferd]

It's something I mostly noticed as a side-effect when developing changes to the compiler, and running bootstrap would pick up files from the previous bootstrap rather than the current up-to-date files.

I figure there would be some amount of safety in just making clean builds with vendored dependencies.

The issue with the bootstrap process is that it has to be portable but has access to no compiled code and no dependencies, so it needs to do things like fetch certifi bundles without validating them in the safest way -- bootstrapping is one of the least safe phases of deployment as it is; an offline mode would be nice, but to be safest it needs to already have access to deps to fetch files. Until Erlang/OTP has a safe way to access the host's cert store in a portable manner, I sort of consider it safest to just use rebar3 escriptize rather than bootstrap when possible, since it will at least reuse the certifi bundle from a previous build.

One thing you may need on top of the "don't go fetching these things" is the steps mentioned in #1281 to bypass checks. The bootstrap process is minimal (we build enough to be able to call rebar3 on itself) so a proper full build will need to be able to run offline on normal rebar3 as well. That will include the pre-fetched dependencies, but also having a pre-cached hex index.

In the long run our objective would be to get rid of dependencies and fold them back into rebar3, but there's no established timeline for that.

[zuiderkwast]

I tried this and it already works without modifications:

1. ./bootstrap with Internet access

2. Turn off WiFi

3. ./bootstrap. It succeeds and the deps are not deleted. It prints...

   Dependency providers already exists
   Dependency getopt already exists
   Dependency cf already exists
   Dependency erlware_commons already exists
   Dependency certifi already exists
   ...
   

Then, I tried to rm -r _build/bootstrap _build/prod _build/default/lib/*/ebin/* before bootstrapping without Internet access. It succeeds. It seems that the deps can just be placed under _build/default/lib prior to bootstrapping and they'll be used.

[zuiderkwast]

I've created #2642 which just copies deps from _checkouts to _build/.../.... It's a first step. I hope it's acceptable. :-)

I think an offline mode, e.g. an environment variable that is checked every time httpc is called and prevents it from fetching anything, would be a next step. WDYT?

[zuiderkwast]

Solved by #2720?

[ferd]

Yep, that was the final intended step of this. Now it should be possible to have offline builds without abusing checkouts (in umbrella projects)