Enhance cross validation

[manuelgitgomes]

Enhance cross validation by using scikit-learn functions.

[manuelgitgomes]

Hello @miguelriemoliveira and @Kazadhum.

I have now added cross validation support in batch execution on the branch dev/cross-validation.

The user now defines type of cross validation and its parameters in data.yaml:

atom/atom_batch_execution/experiments/rrbot_example/data.yml

Lines 23 to 26 in b61df4d

	cross_validation:
	type: "stratified-k-fold"
	n_splits: 3 # Number of folds
	train_size: # Percentage of the dataset used for training, only used in StratifiedShuffleSplit

Right now, fold creation is supported with StratifiedKFold, KFold, LeaveOneOut, and StratifiedShuffleSplit from scikit-learn.
The classes used for stratification are the combination of sensors and patterns detected, as written in:

atom/atom_batch_execution/scripts/batch_execution

Lines 30 to 54 in b61df4d

	def generateClasses(dataset):
	"""
	Generate classes based on an ATOM dataset.
	Classes follow the format detected_pattern--detected_sensor1-detected_sensor2-[...]---[...]
	Args:
	dataset (dict): ATOM dataset.
	Returns:
	tuple: A tuple containing the classes and collection keys.
	"""
	classes = []
	collection_keys = list(dataset['collections'].keys())
	for collection_key in collection_keys:
	detected_sensors = []
	class_name = ''
	for pattern_key in dataset['patterns'].keys():
	for sensor_key in dataset['sensors'].keys():
	if dataset['collections'][collection_key]['labels'][pattern_key][sensor_key]['detected']:
	detected_sensors.append(sensor_key)
	detected_pattern_and_sensors = pattern_key + '--' + '-'.join(detected_sensors)
	class_name += detected_pattern_and_sensors + '---'
	classes.append(class_name.rstrip('---'))
	return classes, collection_keys

This then creates a new auto_rendered.yaml, with division of each run in folds, using -csf to define the collections used by the fold:

atom/atom_batch_execution/experiments/rrbot_example/auto_rendered.yaml

Lines 27 to 44 in b61df4d

	nig_0.1_run001_fold001:
	cmd: |
	rosrun atom_calibration calibrate -json $ATOM_DATASETS/rrbot/train/dataset.json \
	-v -max_nfev 2 -ss 1 \
	-nig 0.1 0.1 \
	-csf 'lambda x: int(x) in [0, 3]' \
	&& \
	rosrun atom_evaluation rgb_to_rgb_evaluation \
	-train_json $ATOM_DATASETS/rrbot/train/atom_calibration.json \
	-test_json $ATOM_DATASETS/rrbot/train/dataset.json \
	-ss rgb_left -st rgb_right \
	-csf 'lambda x: int(x) in [1, 2]' \
	-sfr -sfrn /tmp/rgb_rgb_evaluation.csv
	files_to_collect:
	- '$ATOM_DATASETS/rrbot/train/atom_calibration.json'
	- '$ATOM_DATASETS/rrbot/train/atom_calibration_params.yml'
	- '$ATOM_DATASETS/rrbot/train/command_line_args.yml'
	- '/tmp/rgb_rgb_evaluation.csv'

process_results is also adapted to run with these folds!

I have done a test with rrbot and everything seemed nice, can you test on your machines?

[miguelriemoliveira]

Hi @manuelgitgomes ,

looks great. Thanks.

@Kazadhum I do not have a lot of time right now. Can you test it please?

One question: if we want the run the old way, is it possible or not?

[Kazadhum]

Hi@manuelgitgomes and @miguelriemoliveira! I'll test it as soon as I can. I'll try to tell you something today.

[Kazadhum]

Hi @manuelgitgomes and @miguelriemoliveira !
I just tested with stratified k-folding and it seems to be working correctly at first glance, as well as process_results.

BTW, is it still the case that it doesn't make sense to have the collection column in the processed results, since it also averages the collection number?

I can run more thorough tests next week, if necessary

[manuelgitgomes]

One question: if we want the run the old way, is it possible or not?

Right now, you can't, but I can change that easily enough (I think). On it.

BTW, is it still the case that it doesn't make sense to have the collection column in the processed results, since it also averages the collection number?

It doesn't, I believe. I can delete it.

[Kazadhum]

BTW, is it still the case that it doesn't make sense to have the collection column in the processed results, since it also averages the collection number?

It doesn't, I believe. I can delete it.

Now that I mention it, I think it doesn't make much sense to have any lines at all besides the "Average" line, since all other values are "meaningless" (as they belong to different collections). Do you agree?

[manuelgitgomes]

Now that I mention it, I think it doesn't make much sense to have any lines at all besides the "Average" line, since all other values are "meaningless" (as they belong to different collections). Do you agree?

In the processed results? Sure, makes sense. Can also be removed, but I don't see any wrong in leaving it there, right?

[manuelgitgomes]

Changed batch execution and process results to allow for an empty cross validation definition in data.yml.
This guarantees backwards compatibility, which was tested with old_template.yml.j2
Also removed the "Collection #" column from the processed results.

[miguelriemoliveira]

Looks great!

[manuelgitgomes]

This has been merged to main, closing.