_get_cv_score update

[canberkboun9]

Check: https://scikit-learn.org/1.5/modules/generated/sklearn.model_selection.cross_validate.html

I have never used lofo before. I tried it today. I tried using sklearn 1.4.0. Then I encountered other errors, but it's another story.

For now, the bug resulting from the 1.5 and 1.6 should be looked at.

Sorry for any convention breaking issue format, if any.

[aerdem4]

Thank for the heads up, I will patch it.

Then I encountered other errors, but it's another story.

What else did you encounter?

[canberkboun9]

I wanted to see if sklearn 1.4.0 would work, as I said.

I am working on the home credit default risk dataset on kaggle

I think the error is caused by me having done zero preprocessing the data. I don't know whether this is about another bug. It may be my mistake. On the other hand, I get the same error when I try using a preprocessed data.

Error Message:

---

ValueError Traceback (most recent call last)
in <cell line: 0>()
----> 1 importances = fi.get_importance()

4 frames
/usr/local/lib/python3.11/dist-packages/lofo/lofo_importance.py in get_importance(self)
71 Dataframe with feature names and corresponding importance mean and std (sorted by importance)
72 """
---> 73 base_cv_score = self._get_cv_score(feature_to_remove=None)
74 feature_list = self.dataset.feature_names + list(self.dataset.feature_groups.keys())
75

/usr/local/lib/python3.11/dist-packages/lofo/lofo_importance.py in _get_cv_score(self, feature_to_remove)
55 with warnings.catch_warnings():
56 warnings.simplefilter("ignore")
---> 57 cv_results = cross_validate(self.model, X, y, cv=self.cv, scoring=self.scoring, fit_params=fit_params, groups=self.cv_groups)
58 return cv_results['test_score']
59

/usr/local/lib/python3.11/dist-packages/sklearn/utils/_param_validation.py in wrapper(*args, **kwargs)
211 )
212 ):
--> 213 return func(*args, **kwargs)
214 except InvalidParameterError as e:
215 # When the function is just a wrapper around an estimator, we allow

/usr/local/lib/python3.11/dist-packages/sklearn/model_selection/_validation.py in cross_validate(estimator, X, y, groups, scoring, cv, n_jobs, verbose, fit_params, params, pre_dispatch, return_train_score, return_estimator, return_indices, error_score)
443 )
444
--> 445 _warn_or_raise_about_fit_failures(results, error_score)
446
447 # For callable scoring, the return type is only know after calling. If the

/usr/local/lib/python3.11/dist-packages/sklearn/model_selection/_validation.py in _warn_or_raise_about_fit_failures(results, error_score)
529 f"Below are more details about the failures:\n{fit_errors_summary}"
530 )
--> 531 raise ValueError(all_fits_failed_message)
532
533 else:

ValueError:
All the 4 fits failed.
It is very likely that your model is misconfigured.
You can try to debug the error by setting error_score='raise'.

Below are more details about the failures:

4 fits failed with the following error:
Traceback (most recent call last):
File "_catboost.pyx", line 2547, in _catboost.get_float_feature
File "_catboost.pyx", line 1226, in _catboost._FloatOrNan
File "_catboost.pyx", line 1021, in _catboost._FloatOrNanFromString
TypeError: Cannot convert 'b'Cash loans'' to float

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.11/dist-packages/sklearn/model_selection/_validation.py", line 890, in _fit_and_score
estimator.fit(X_train, y_train, **fit_params)
File "/usr/local/lib/python3.11/dist-packages/catboost/core.py", line 5245, in fit
self._fit(X, y, cat_features, text_features, embedding_features, None, graph, sample_weight, None, None, None, None, baseline, use_best_model,
File "/usr/local/lib/python3.11/dist-packages/catboost/core.py", line 2395, in _fit
train_params = self._prepare_train_params(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/catboost/core.py", line 2275, in _prepare_train_params
train_pool = _build_train_pool(X, y, cat_features, text_features, embedding_features, pairs, graph,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/catboost/core.py", line 1513, in _build_train_pool
train_pool = Pool(X, y, cat_features=cat_features, text_features=text_features, embedding_features=embedding_features, pairs=pairs, graph=graph, weight=sample_weight, group_id=group_id,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/catboost/core.py", line 855, in init
self._init(data, label, cat_features, text_features, embedding_features, embedding_features_data, pairs, graph, weight,
File "/usr/local/lib/python3.11/dist-packages/catboost/core.py", line 1491, in _init
self._init_pool(data, label, cat_features, text_features, embedding_features, embedding_features_data, pairs, graph, weight,
File "_catboost.pyx", line 4339, in _catboost._PoolBase._init_pool
File "_catboost.pyx", line 4409, in _catboost._PoolBase._init_pool
File "_catboost.pyx", line 4310, in _catboost._PoolBase._init_objects_order_layout_pool
File "_catboost.pyx", line 3802, in _catboost._set_data
File "_catboost.pyx", line 3785, in _catboost._set_data_from_generic_matrix
File "_catboost.pyx", line 2549, in _catboost.get_float_feature
_catboost.CatBoostError: Bad value for num_feature[non_default_doc_idx=0,feature_idx=1]="Cash loans": Cannot convert 'b'Cash loans'' to float

[aerdem4]

Thanks for sharing. The issue seems to be not related to lofo. What is the data type of your "Cash loans" feature?

[venturero]

Hi @aerdem4 & @canberkboun9 ,

I am using LOFO Importance on the Home Credit Default Risk dataset. Initially, I encountered issues with scikit-learn version 1.6, but it worked with version 1.4.

However, the LOFO Importance code is taking several hours to run. Do you have any suggestions or recommendations for improving its execution speed?

Thanks

[venturero]

Hi again,

I've worked a lot on the code. The latency was caused by the number of folds and features. I reduced both, and the code now runs much faster—in minutes instead of hours.

[aerdem4]

The issue must be resolved now (version 0.3.5). Please update the package and have a look.

[venturero]

Yes it worked. Thanks a lot.