MXNet GPU build on CPU machine fails

[vrakesh]

Post PR #15449 , when trying to build mxnet with GPU support on a CPU instance fails, with the following error

Running: OpWrapperGenerator.py
terminate called after throwing an instance of 'dmlc::Error'
what(): [00:10:08]include/mxnet/base.h:427: Check failed: e == cudaSuccess (35 vs. 0) : CUDA: CUDA driver version is insufficient for CUDA runtime version
Stack trace:
[bt] (0) build/lib/libmxnet.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x25) [0x7fcf63bb2793]
[bt] (1) build/lib/libmxnet.so(mxnet::Context::GetGPUCount()+0x11b) [0x7fcf66d39325]
[bt] (2) build/lib/libmxnet.so(+0x5ebcfaa) [0x7fcf66df5faa]
[bt] (3) /build/lib/libmxnet.so(+0x5ebd180) [0x7fcf66df6180]
[bt] (4) build/lib/libmxnet.so(+0x5ebd1b7) [0x7fcf66df61b7]
[bt] (5) /lib64/ld-linux-x86-64.so.2(+0xe9cf) [0x7fcfd1f419cf]
[bt] (6) /lib64/ld-linux-x86-64.so.2(+0x131f5) [0x7fcfd1f461f5]
[bt] (7) /lib64/ld-linux-x86-64.so.2(+0xe5e6) [0x7fcfd1f415e6]
[bt] (8) /lib64/ld-linux-x86-64.so.2(+0x1299a) [0x7fcfd1f4599a]

The cmake setting is as follows

 cmake \
    -DUSE_OPENCV=${USE_OPENCV:-ON}\
    -DUSE_OPENMP=ON\
    -DUSE_CUDA=${USE_CUDA:-ON}\
    -DUSE_CUDNN=${USE_CUDA:-ON}\
    -DUSE_LAPACK=OFF\
    -DUSE_MKL_IF_AVAILABLE=ON\
    -DUSE_MKLML_MKL=OFF\
    -DUSE_MKLDNN=${USE_MKLDNN:-ON}\
    -DUSE_SSE=ON\
    -DUSE_GPERFTOOLS=OFF\
    -DUSE_JEMALLOC=${USE_JEMALLOC:-ON}\
    -DUSE_PROFILER=${USE_PROFILER:-ON}\
    -DUSE_DIST_KVSTORE=${USE_DIST_KVSTORE:-ON}\
    -DUSE_CPP_PACKAGE=ON\
    -DCMAKE_BUILD_TYPE=$CMAKE_BUILD_TYPE \
    -DCUDA_ARCH_NAME=Manual\
    -DCUDA_ARCH_BIN=37,70,75\
    -DCUDA_ARCH_PTX=70,75 \
    -DINSTALL_PYTHON_VERSIONS="${PYTHON_VERSIONS:-2.7;3.4;3.5;3.6}"\
    -DENABLE_CUDA_RTC=${ENABLE_CUDA_RTC:-OFF}\
    -DUSE_SIGNAL_HANDLER=OFF\
    -DUSE_NCCL=${USE_NCCL:-OFF}     

cmake cpp_package_op_h

[mxnet-label-bot]

Hey, this is the MXNet Label Bot.
Thank you for submitting the issue! I will try and suggest some labels so that the appropriate MXNet community members can help resolve it.
Here are my recommended labels: Build

[vrakesh]

@mxnet-label-bot add [Build]

[DickJC123]

I have just today seen something similar on our CI systems and have a fix. I'll submit a quick PR. The idea is that on systems with no cuda driver, cudaDriverGetVersion(&driver_version) will show a driver_version of 0. I can bypass the cuda/cudnn checks and any request for a GPUCount in that case.

[vrakesh]

@DickJC123 Thanks a lot for quick PR fix

[vrakesh]

Can confirm #15551 fixed this. closing issue

[KellenSunderland]

Thanks for the quick fix @DickJC123

[zachgk]

@DickJC123 Can you take a look at another case of this? We run a nightly maven snapshot pipeline in the CI that builds for GPU on a CPU machine as well. However, it now seems to be failing some Scala unit tests that are run (still on the CPU machine while building for GPU). It fails either when trying to create a KVStore or use a DataIterator (MXDataIterNext).

The error message it fails with is (from http://jenkins.mxnet-ci.amazon-ml.com/job/restricted-publish-artifacts/job/master/208/):

org.apache.mxnet.MXNetError: [20:13:12] /work/mxnet/3rdparty/dmlc-core/include/dmlc/threadediter.h:472: [20:13:12] include/mxnet/base.h:459: Check failed: e == cudaSuccess (30 vs. 0) :  CUDA: unknown error
Stack trace:
  [bt] (0) /work/mxnet/scala-package/native/target/libmxnet.so(+0x54430b) [0x7f95a73f130b]
  [bt] (1) /work/mxnet/scala-package/native/target/libmxnet.so(+0x2a8d4a5) [0x7f95a993a4a5]
  [bt] (2) /work/mxnet/scala-package/native/target/libmxnet.so(+0x346eee5) [0x7f95aa31bee5]
  [bt] (3) /work/mxnet/scala-package/native/target/libmxnet.so(+0x9395bd) [0x7f95a77e65bd]
  [bt] (4) /work/mxnet/scala-package/native/target/libmxnet.so(+0x2c63b3f) [0x7f95a9b10b3f]
  [bt] (5) /work/mxnet/scala-package/native/target/libmxnet.so(+0x2c0a564) [0x7f95a9ab7564]
  [bt] (6) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x91c10) [0x7f979d4e7c10]
  [bt] (7) /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184) [0x7f979e671184]
  [bt] (8) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f979eda203d]

It builds MXNet using the script at https://github.com/apache/incubator-mxnet/blob/master/tools/staticbuild/build.sh and sets up cuda for building using https://github.com/apache/incubator-mxnet/blob/master/tools/setup_gpu_build_tools.sh.