Unable to Train DeepEdit-DynUNet from Scratch on BraTS-2018 Dataset

[ArashDadras]

I am attempting to train a DeepEdit-DynUNet model from scratch on the BraTS-2018 dataset for segmentation tasks. However, I have been unsuccessful so far. I followed the tutorial (here), and modified the DeepEdit configuration file as follows:

self.labels = {
     "background": 0,
     "edema": 1,
      "tumor core": 2,
     "enhancing tumor": 3
}

# Number of input channels - 4 for BRATS and 1 for spleen
self.number_intensity_ch = 4

network = self.conf.get("network", "dynunet")

# Model Files
 self.path = [
    os.path.join(self.model_dir, f"pretrained_{self.name}_{network}.pt"),  # pretrained
    os.path.join(self.model_dir, f"{self.name}_{network}.pt"),  # published
]

# Download PreTrained Model
if strtobool(self.conf.get("use_pretrained_model", "false")):
   url = f"{self.conf.get('pretrained_path', self.PRE_TRAINED_PATH)}"
   url = f"{url}/radiology_deepedit_{network}_multilabel.pt"
   download_file(url, self.path[0])

I structured the training image folder as described in the tutorial. For testing purposes, I copied some of my labels into the labels/final folder to perform an initial training run.

After starting the server and opening 3D Slicer, I initiated the training process. However, I encountered the following error:IndexError: index 4 is out of bounds for dimension 0 with size 4

I have successfully used the BraTS dataset with a segmentation model using the same approach, but with different configurations (only changing the labels and setting the input channel number to 4). However, with the Deepedit model, I am unable to run the training at all.

Questions:

Is there something I am missing in the configuration?

Could the issue be related to the dataset structure or the way labels are being handled?

Any guidance or suggestions would be greatly appreciated. Thank you in advance!

Full Error Log:
Traceback (most recent call last): File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\ignite\engine\engine.py", line 1068, in _run_once_on_dataset_as_gen self.state.output = self._process_function(self, self.state.batch) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\apps\deepedit\interaction.py", line 100, in __call__ return engine._iteration(engine, batchdata) # type: ignore[arg-type] File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\engines\trainer.py", line 281, in _iteration engine.fire_event(IterationEvents.MODEL_COMPLETED) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\ignite\engine\engine.py", line 449, in fire_event return self._fire_event(event_name) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\ignite\engine\engine.py", line 425, in _fire_event func(*first, *(event_args + others), **kwargs) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\engines\workflow.py", line 225, in _run_postprocessing engine.state.batch[i], engine.state.output[i] = engine_apply_transform(b, o, posttrans) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\engines\utils.py", line 335, in engine_apply_transform transformed_data = apply_transform(transform, data) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\transforms\transform.py", line 171, in apply_transform raise RuntimeError(f"applying transform {transform}") from e RuntimeError: applying transform <monai.transforms.compose.Compose object at 0x000001F242E2AC10> [2025-03-12 15:04:18.681][ERROR](ignite.engine.engine.SupervisedTrainer) - Current run is terminating due to exception: applying transform <monai.transforms.compose.Compose object at 0x00000261DF43D700> [2025-03-12 15:04:18.689][ERROR](ignite.engine.engine.SupervisedTrainer) - Engine run is terminating due to exception: applying transform <monai.transforms.compose.Compose object at 0x00000261DF43D700> [2025-03-12 15:04:18.699][ERROR](ignite.engine.engine.SupervisedTrainer) - Engine run is terminating due to exception: applying transform <monai.transforms.compose.Compose object at 0x000001F242E2AC10> 2025-03-12 15:04:18,699 - ERROR - Exception: applying transform <monai.transforms.compose.Compose object at 0x000001F242E2AC10> Traceback (most recent call last): File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\transforms\transform.py", line 141, in apply_transform return _apply_transform(transform, data, unpack_items, lazy, overrides, log_stats) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\transforms\transform.py", line 98, in _apply_transform return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\apps\deepedit\transforms.py", line 752, in __call__ d[f"pred_{key_label}"] = d[key][idx + 1, ...][None] File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\data\meta_tensor.py", line 282, in __torch_function__ ret = super().__torch_function__(func, types, args, kwargs) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\torch\_tensor.py", line 1648, in __torch_function__ ret = func(*args, **kwargs) IndexError: index 4 is out of bounds for dimension 0 with size 4 The above exception was the direct cause of the following exception: Traceback (most recent call last): File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\transforms\transform.py", line 141, in apply_transform return _apply_transform(transform, data, unpack_items, lazy, overrides, log_stats) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\transforms\transform.py", line 98, in _apply_transform return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\transforms\compose.py", line 335, in __call__ result = execute_compose( File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\transforms\compose.py", line 111, in execute_compose data = apply_transform( File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\transforms\transform.py", line 171, in apply_transform raise RuntimeError(f"applying transform {transform}") from e RuntimeError: applying transform <monai.apps.deepedit.transforms.SplitPredsLabeld object at 0x000001F242E2AD00> The above exception was the direct cause of the following exception: Traceback (most recent call last): File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\ignite\engine\engine.py", line 959, in _internal_run_as_gen epoch_time_taken += yield from self._run_once_on_dataset_as_gen() File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\ignite\engine\engine.py", line 1087, in _run_once_on_dataset_as_gen self._handle_exception(e) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\ignite\engine\engine.py", line 636, in _handle_exception self._fire_event(Events.EXCEPTION_RAISED, e) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\ignite\engine\engine.py", line 425, in _fire_event func(*first, *(event_args + others), **kwargs) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\handlers\stats_handler.py", line 201, in exception_raised raise e File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\ignite\engine\engine.py", line 1068, in _run_once_on_dataset_as_gen self.state.output = self._process_function(self, self.state.batch) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\apps\deepedit\interaction.py", line 100, in __call__ return engine._iteration(engine, batchdata) # type: ignore[arg-type] File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\engines\trainer.py", line 281, in _iteration engine.fire_event(IterationEvents.MODEL_COMPLETED) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\ignite\engine\engine.py", line 449, in fire_event return self._fire_event(event_name) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\ignite\engine\engine.py", line 425, in _fire_event func(*first, *(event_args + others), **kwargs) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\engines\workflow.py", line 225, in _run_postprocessing engine.state.batch[i], engine.state.output[i] = engine_apply_transform(b, o, posttrans) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\engines\utils.py", line 335, in engine_apply_transform transformed_data = apply_transform(transform, data) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\transforms\transform.py", line 171, in apply_transform raise RuntimeError(f"applying transform {transform}") from e RuntimeError: applying transform <monai.transforms.compose.Compose object at 0x000001F242E2AC10> W0312 15:04:19.793660 2596 monaienv\Lib\site-packages\torch\multiprocessing\spawn.py:169] Terminating process 5636 via signal SIGTERM Traceback (most recent call last): File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\runpy.py", line 87, in _run_code exec(code, run_globals) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monailabel\interfaces\utils\app.py", line 128, in <module> run_main() File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monailabel\interfaces\utils\app.py", line 113, in run_main result = a.train(request) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monailabel\interfaces\app.py", line 423, in train result = task(request, self.datastore()) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monailabel\tasks\train\basic_train.py", line 466, in __call__ torch.multiprocessing.spawn(main_worker, nprocs=world_size, args=(world_size, req, datalist, self)) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\torch\multiprocessing\spawn.py", line 340, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method="spawn") File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\torch\multiprocessing\spawn.py", line 296, in start_processes while not context.join(): File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\torch\multiprocessing\spawn.py", line 215, in join raise ProcessRaisedException(msg, error_index, failed_process.pid) torch.multiprocessing.spawn.ProcessRaisedException: -- Process 0 terminated with the following error: Traceback (most recent call last): File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\transforms\transform.py", line 141, in apply_transform return _apply_transform(transform, data, unpack_items, lazy, overrides, log_stats) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\transforms\transform.py", line 98, in _apply_transform return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\apps\deepedit\transforms.py", line 752, in __call__ d[f"pred_{key_label}"] = d[key][idx + 1, ...][None] File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\data\meta_tensor.py", line 282, in __torch_function__ ret = super().__torch_function__(func, types, args, kwargs) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\torch\_tensor.py", line 1648, in __torch_function__ ret = func(*args, **kwargs) IndexError: index 4 is out of bounds for dimension 0 with size 4 The above exception was the direct cause of the following exception: Traceback (most recent call last): File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\transforms\transform.py", line 141, in apply_transform return _apply_transform(transform, data, unpack_items, lazy, overrides, log_stats) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\transforms\transform.py", line 98, in _apply_transform return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\transforms\compose.py", line 335, in __call__ result = execute_compose( File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\transforms\compose.py", line 111, in execute_compose data = apply_transform( File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\transforms\transform.py", line 171, in apply_transform raise RuntimeError(f"applying transform {transform}") from e RuntimeError: applying transform <monai.apps.deepedit.transforms.SplitPredsLabeld object at 0x000001F242E2AD00> The above exception was the direct cause of the following exception: Traceback (most recent call last): File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\torch\multiprocessing\spawn.py", line 90, in _wrap fn(i, *args) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monailabel\tasks\train\basic_train.py", line 735, in main_worker task.train(rank, world_size, request, datalist) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monailabel\tasks\train\basic_train.py", line 559, in train context.trainer.run() File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\engines\trainer.py", line 55, in run super().run() File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\engines\workflow.py", line 282, in run super().run(data=self.data_loader, max_epochs=self.state.max_epochs) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\ignite\engine\engine.py", line 892, in run return self._internal_run() File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\ignite\engine\engine.py", line 935, in _internal_run return next(self._internal_run_generator) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\ignite\engine\engine.py", line 993, in _internal_run_as_gen self._handle_exception(e) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\ignite\engine\engine.py", line 636, in _handle_exception self._fire_event(Events.EXCEPTION_RAISED, e) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\ignite\engine\engine.py", line 425, in _fire_event func(*first, *(event_args + others), **kwargs) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\handlers\stats_handler.py", line 201, in exception_raised raise e File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\ignite\engine\engine.py", line 959, in _internal_run_as_gen epoch_time_taken += yield from self._run_once_on_dataset_as_gen() File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\ignite\engine\engine.py", line 1087, in _run_once_on_dataset_as_gen self._handle_exception(e) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\ignite\engine\engine.py", line 636, in _handle_exception self._fire_event(Events.EXCEPTION_RAISED, e) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\ignite\engine\engine.py", line 425, in _fire_event func(*first, *(event_args + others), **kwargs) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\handlers\stats_handler.py", line 201, in exception_raised raise e File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\ignite\engine\engine.py", line 1068, in _run_once_on_dataset_as_gen self.state.output = self._process_function(self, self.state.batch) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\apps\deepedit\interaction.py", line 100, in __call__ return engine._iteration(engine, batchdata) # type: ignore[arg-type] File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\engines\trainer.py", line 281, in _iteration engine.fire_event(IterationEvents.MODEL_COMPLETED) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\ignite\engine\engine.py", line 449, in fire_event return self._fire_event(event_name) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\ignite\engine\engine.py", line 425, in _fire_event func(*first, *(event_args + others), **kwargs) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\engines\workflow.py", line 225, in _run_postprocessing engine.state.batch[i], engine.state.output[i] = engine_apply_transform(b, o, posttrans) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\engines\utils.py", line 335, in engine_apply_transform transformed_data = apply_transform(transform, data) File "C:\Users\AIMLab2\.conda\envs\monaienv\lib\site-packages\monai\transforms\transform.py", line 171, in apply_transform raise RuntimeError(f"applying transform {transform}") from e RuntimeError: applying transform <monai.transforms.compose.Compose object at 0x000001F242E2AC10> [2025-03-12 15:04:20,382] [19196] [ThreadPoolExecutor-0_0] [INFO] (monailabel.utils.async_tasks.utils:83) - Return code: 1