Parallel build support

[mattgodbolt]

This might be a big ask; and may be a bazel core project issue; but when consuming larger projects that need to be configure/make'd (e.g. OpenSSL, postgres, hdf5...) it would be great to be able to use parallel builds.

Of course, pragmatically one might override make_commands to have make -j12 in it...but that has disastrous consequences with other parallel builds (not in the least also on machines with fewer than 12 cores...)

In the land of gumdrops and unicorns, one could imagine bazel being able to trick out make by passing it the same magic that $(MAKE) or +make does in a real makefile, along with a -j set from the --jobs setting. That way, the make process will play nicely with the bazel build.

I realise this is a big ask: but if one doesn't ask, then maybe it won't happen! Any ideas on improving this in general also welcomed!

Thanks all!

[mattgodbolt]

In case I was being incoherent, the POSIX jobserver for make is described in more detail here: https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#POSIX-Jobserver

[irengrig]

Thank you for the feedback, I think it is worth doing.
The other problem is how to balance the number of Bazel's jobs with make's jobs (also, there may be several parallel rules_foreign_cc targets being processed.)

[HackAttack]

I’d argue that this is vital to make this project usable. For example, running the example on my machine and building OpenBLAS takes 6 minutes, but when I manually build it with make it takes 13 seconds!

[lawsonAGMT]

I will second this. We have a cmake project dependency that is significantly larger than any other component.

[rdelfin]

I'm finding myself with a cmake project whose build time goes up by almost 4x when compared with how it would go with a manual build. As a stop gap, what would be needed to simply add a fixed parameter in the target that lets you specify how many processes you want to create for the job server?

[HackAttack]

Related:
bazelbuild/bazel#6477
bazelbuild/bazel#10443

[dhbaird]

Not a solution yet: But if I could find a way to let cmake_external inherit file descriptors, then a wrapper Makefile would hypothetically get me to a usable workaround, where CPU over-commit would have a 1x upper bound:

build: ; +bazel @all//... --action_env="MAKEFLAGS=${MAKEFLAGS}"
(where likely MAKEFLAGS=" -j --jobserver-fds=3,4" and so fds 3 and 4 need to be inherited by cmake_external)

Better yet, as mentioned above, would be to have Bazel itself implement the Make jobserver. Maybe this could be inspiration:
https://github.com/olsner/jobclient "GNU make jobserver and client for e.g. shell scripts"

[dhbaird]

Following up from my previous comment- came up with this really sketchy workaround. Can't say I'm proud of it. But it "works," and puts a reasonable bound on CPU over-subscription without blocking parallelism.

1. Create a home for jobserver fifos to live:

mkdir $HOME/.jobserver

# Create jmake (to use later instead of make):
cat > ~/.jobserver/jmake << EOF
#!/bin/bash
# Note: must expand $HOME here because Bazel removes $HOME from environment:
RD=$HOME/.jobserver/rd
WR=$HOME/.jobserver/wr 
exec 3<\$RD
exec 4>\$WR
MAKEFLAGS=" -j --jobserver-fds=3,4" make "\$@"
EOF
chmod a+rx ~/.jobserver/jmake

2. Update make_commands in your BUILD file:

cmake_external(
    ...
    make_commands = ["/path/to/home/.jobserver/jmake", "/path/to/home/.jobserver/jmake install"]
)

3. Start a jobserver and run Bazel (adjust NPROCS to whatever bound you want):

( NPROCS=32 ;
  RD=$HOME/.jobserver/rd ;
  WR=$HOME/.jobserver/wr ;
  rm -f $RD $WR ;
  mkfifo $RD $WR ;
  echo "jobserver: ; +( cat >$RD <&3 & ( while true; do cat <$WR; sleep 0.1; done ) >&4 )" | \
      make -f - -j ${NPROCS} ) &

bazel build @all//... --sandbox_writable_path=$HOME/.jobserver

[jsharpe]

You could skip step 2 by creating a custom make toolchain for make that uses your make script.

[dhbaird]

Implemented @jsharpe's feedback (thanks!) and cleaned things up a bit, and then I turned it into a whole project that I swear I wasn't planning to do just one day ago. There is a questionable dependency on the "master" (or "main") branch of rules_foreign_cc that I'm not sure how to reconcile. Nevertheless here it is,

https://github.com/dhbaird/rules_foreign_cc_jobserver

[slsyy]

Calling make with an option --load-average can be a good workaround e.g:

NUMBER_OF_CPUS= "$(grep -c processor /proc/cpuinfo)"
make -j "$NUMBER_OF_CPUS"  -l "$NUMBER_OF_CPUS"

[fuhailin]

Calling make with an option --load-average can be a good workaround e.g:

NUMBER_OF_CPUS= "$(grep -c processor /proc/cpuinfo)"
make -j "$NUMBER_OF_CPUS"  -l "$NUMBER_OF_CPUS"

that sounds like a good concise solution, how should i integrate your solution with rules_foreign_cc build script?
Could you please an example?
And what about Macos?

[slsyy]

@fuhailin I don't know, /proc/cpuinfo was just a quick thought. You can specifiy flags in bazel command to limit cpu like --local_cpu_resources or --jobs to reduce number of concurrent tasks. I guess it would be best to extract these information from the bazel itself

In my current build I just use hardcoded values like:

make -j 12 -l 12

[UebelAndre]

You could also use something like

make(
    name = "make_lib",
    env = {
        "CLANG_WRAPPER": "$(execpath //make_simple/code:clang_wrapper.sh)",
    },
    lib_source = "//make_simple/code:srcs",
    make_commands = [
        "make -j `nproc`",
        "make install",
    ],
    static_libraries = ["liba.a"],
    tools_deps = ["//make_simple/code:clang_wrapper.sh"],
)

Which is a slight alteration to:
https://github.com/bazelbuild/rules_foreign_cc/blob/175b29c6f78cf3c78516836587c268f3d0690526/examples/make_simple/BUILD.bazel#L4-L12