BUG in GPU histogram

[lorenzoridolfi]

Environment info

Operating System: Fedora 26
CPU: I5
GPU: NVidia GTX 1060
C++/Python/R version:
Python 3.6.2
Cuda 9.0

Error Message:

[LightGBM] [Info] Number of positive: 17355, number of negative: 458814
[LightGBM] [Warning] Only find one worker, will switch to serial tree learner.
[LightGBM] [Info] This is the GPU trainer!!
[LightGBM] [Info] Total Bins 1357
[LightGBM] [Info] Number of data: 476169, number of used features: 57
[LightGBM] [Info] Using GPU Device: GeForce GTX 1060 6GB, Vendor: NVIDIA Corporation
[LightGBM] [Info] Compiling OpenCL Kernel with 256 bins...
[LightGBM] [Info] GPU programs have been built
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 43 dense feature groups (19.98 MB) transfered to GPU in 0.048936 secs. 9 sparse feature groups.
[LightGBM] [Info] Number of positive: 17355, number of negative: 458814
[LightGBM] [Warning] Only find one worker, will switch to serial tree learner.
[LightGBM] [Info] This is the GPU trainer!!
[LightGBM] [Info] Total Bins 1357
[LightGBM] [Info] Number of data: 476169, number of used features: 57
[LightGBM] [Info] Using GPU Device: GeForce GTX 1060 6GB, Vendor: NVIDIA Corporation
[LightGBM] [Info] Compiling OpenCL Kernel with 256 bins...
[LightGBM] [Info] GPU programs have been built
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 43 dense feature groups (19.98 MB) transfered to GPU in 0.048049 secs. 9 sparse feature groups.
[LightGBM] [Info] Number of positive: 17355, number of negative: 458814
[LightGBM] [Warning] Only find one worker, will switch to serial tree learner.
[LightGBM] [Info] This is the GPU trainer!!
[LightGBM] [Info] Total Bins 1357
[LightGBM] [Info] Number of data: 476169, number of used features: 57
[LightGBM] [Info] Using GPU Device: GeForce GTX 1060 6GB, Vendor: NVIDIA Corporation
[LightGBM] [Info] Compiling OpenCL Kernel with 256 bins...
[LightGBM] [Info] GPU programs have been built
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 43 dense feature groups (19.98 MB) transfered to GPU in 0.039569 secs. 9 sparse feature groups.
[LightGBM] [Info] Number of positive: 17355, number of negative: 458815
[LightGBM] [Warning] Only find one worker, will switch to serial tree learner.
[LightGBM] [Info] This is the GPU trainer!!
[LightGBM] [Info] Total Bins 1357
[LightGBM] [Info] Number of data: 476170, number of used features: 57
[LightGBM] [Info] Using GPU Device: GeForce GTX 1060 6GB, Vendor: NVIDIA Corporation
[LightGBM] [Info] Compiling OpenCL Kernel with 256 bins...
[LightGBM] [Info] GPU programs have been built
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 43 dense feature groups (19.98 MB) transfered to GPU in 0.035209 secs. 9 sparse feature groups.
[LightGBM] [Info] Number of positive: 17356, number of negative: 458815
[LightGBM] [Warning] Only find one worker, will switch to serial tree learner.
[LightGBM] [Info] This is the GPU trainer!!
[LightGBM] [Info] Total Bins 1357
[LightGBM] [Info] Number of data: 476171, number of used features: 57
[LightGBM] [Info] Using GPU Device: GeForce GTX 1060 6GB, Vendor: NVIDIA Corporation
[LightGBM] [Info] Compiling OpenCL Kernel with 256 bins...
[LightGBM] [Info] GPU programs have been built
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 43 dense feature groups (19.98 MB) transfered to GPU in 0.040315 secs. 9 sparse feature groups.
[LightGBM] [Fatal] Bug in GPU histogram! split 8211: 11359, smaller_leaf: 9610, larger_leaf: 9960

Traceback (most recent call last):
File "lightgbm_param.py", line 127, in
main()
File "lightgbm_param.py", line 79, in main
categorical_feature=cat_index_2)
File "/usr/local/anaconda3/lib/python3.6/site-packages/lightgbm/engine.py", line 443, in cv
cvfolds.update(fobj=fobj)
File "/usr/local/anaconda3/lib/python3.6/site-packages/lightgbm/engine.py", line 244, in handlerFunction
ret.append(getattr(booster, name)(*args, **kwargs))
File "/usr/local/anaconda3/lib/python3.6/site-packages/lightgbm/basic.py", line 1436, in update
ctypes.byref(is_finished)))
File "/usr/local/anaconda3/lib/python3.6/site-packages/lightgbm/basic.py", line 48, in _safe_call
raise LightGBMError(_LIB.LGBM_GetLastError())
lightgbm.basic.LightGBMError: b'Bug in GPU histogram! split 8211: 11359, smaller_leaf: 9610, larger_leaf: 9960\n'

Reproducible examples

	params = {
			    'boosting_type': 'gbdt',
			    'objective': 'binary',
			    'metric': 'binary_logloss',
			    'num_leaves': 31,
			    'learning_rate': 0.005,
			    'feature_fraction': 0.9,
			    'bagging_fraction': 0.8,
			    'verbose': 1,
			    'device' : 'gpu'
			}

	d_train = lgb.Dataset(all_x, label=all_y)

	cv_results = lgb.cv(params,
			                d_train,
			                num_boost_round=700,
			                categorical_feature=cat_index_2)

[huanzhang12]

Thanks for reporting this problem! There might be a bug trigger by a race condition in the GPU code. I guess it is related to the feature_fraction and bagging_fraction parameters. Could you please change them to 1.0 and see which parameter causes the problem?

I will also really appreciate if you can reproduce the problem on any public datasets, or share the dataset with me if it is not sensitive. This will greatly help me debug this issue. Thank you!

[lorenzoridolfi]

Hi, setting these two parameters to 1.0 the bug happened, too, but It took several iterations to occur. With the old values the bug happened with very few iterations.

The source code is:
https://www.dropbox.com/s/bqj428pc5vwcpp9/lightgbm_param.py?dl=0

And the data files are:
https://www.dropbox.com/s/6lbpn54sdqn98kd/train.csv?dl=0
https://www.dropbox.com/s/lv8sam3tx415x62/test.csv?dl=0

Best Regards,
Lorenzo

[huanzhang12]

@lorenzoridolfi Thank you for the detailed information on code and data! They are really helpful. I got a little bit busy recently but I will try to catch this bug as quickly as I can.

[lorenzoridolfi]

Any news about this bug? It's almost a month!

Thank you,
Lorenzo

[Laurae2]

ping @huanzhang12 if you have any news

[huanzhang12]

Sorry I got crazily busy recently and did not get a chance to look into this bug. Will try to work on this during thanksgiving holiday. Thanks for your understanding!

[mjaysonnn]

Is this bug related to bin size error? For example when I use GPU-version lgbm

"bin size 16855 cannot run on GPU" error happens.

[guolinke]

@mjaysonnn
GPU version cannot support categorical features with high cardinality.
You can fix it by split one categorical feature into multi categorical features.

[mjmckp]

I am also getting this error, using the latest version of LightGBM:

[LightGBM] [Info] This is the GPU trainer!!
[LightGBM] [Info] Total Bins 165885
[LightGBM] [Info] Number of data: 4561756, number of used features: 658
[LightGBM] [Info] Using GPU Device: GeForce GTX 1080 Ti, Vendor: NVIDIA Corporation
[LightGBM] [Info] Compiling OpenCL Kernel with 256 bins...
[LightGBM] [Info] GPU programs have been built
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (2836.48 MB) transfered to GPU in 1.880998 secs. 7 sparse feature groups
[LightGBM] [Info] Start training from score 0.466854
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.932567 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.892417 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.904232 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.882715 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.908154 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.907432 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.875952 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.865907 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.862585 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.892193 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.891429 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.915810 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.896881 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.894748 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.934688 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.908344 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.879211 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.877866 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.889270 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.850099 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.931063 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.910245 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.856059 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.905356 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.881266 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.867991 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.876403 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.873414 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.899580 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.908517 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.881924 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.907277 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.874490 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.889323 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.890838 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.871581 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.875327 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.885586 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.895304 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.895696 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.935202 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.870557 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.865199 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.912166 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.891637 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.882426 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.894549 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.854855 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.863332 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.881228 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.875864 secs. 7 sparse feature groups
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 651 dense feature groups (1418.24 MB) transfered to GPU in 0.885681 secs. 7 sparse feature groups
[LightGBM] [Fatal] Bug in GPU histogram! split 139388: 62131, smaller_leaf: 62132, larger_leaf: 139387

[huanzhang12]

@mjmckp Could you please provide the dataset and the python/shell script you used to reproduce this error? This will be really helpful for me to debug this issue.

I tried to reproduce the bug with the dataset and code provided by @lorenzoridolfi but I cannot reproduce it on three different machines. I tried different feature_fraction and bagging_fraction values but still cannot make the bug appear. @lorenzoridolfi Could you please try the latest LightGBM and see if you are still encountering the same error?

[mjmckp]

Self-contained repro here: https://www.dropbox.com/sh/9f9u7wm5ithfjbr/AADcQ6k8yDSkA3J3vYqsg4Hta?dl=0

Unzip the file dataset.zip and run lightgbm.exe config=repro.conf, console output is in output.txt.

I am running with:

• LightGBM built from the current master branch (8ce2a232e907d518979e7105842ae575a7427377)
• Windows 10 Professional
• NVidia GTX 1080 Ti

[huanzhang12]

@mjmckp Thank you for providing the dataset and config files! I still cannot reproduce this problem on AMD and NVIDIA GPUs on my machines. However I did observe GPU hang on an Intel integrated GPU, which was not tested thoroughly before.

There might be a bug with max_bin=255. Could you please try to use max_bin=63 and see if this bug still occurs (make sure the log says Compiling OpenCL Kernel with 64 bins). If it disappears, I will investigate the OpenCL kernel for 256 bins carefully.

@mjmckp Another possibility is here: https://github.com/Microsoft/LightGBM/blob/master/src/treelearner/gpu_tree_learner.cpp#L119
If changing max_bin=63 does not work, could you please also try uncomment this line (return 0;) to make GetNumWorkgroupsPerFeature return 0?