Using PYTHONPATH=/home/jun/anaconda3/envs: [2024-11-30 07:01:55,969] [3310] [MainThread] [INFO] (__main__:285) - USING:: version = False [2024-11-30 07:01:55,969] [3310] [MainThread] [INFO] (__main__:285) - USING:: app = /home/jun/iota/0-train_cleaned/LUNGND/apps/radiology [2024-11-30 07:01:55,970] [3310] [MainThread] [INFO] (__main__:285) - USING:: studies = /home/jun/iota/0-train_cleaned/LUNGND/datasets/spine [2024-11-30 07:01:55,970] [3310] [MainThread] [INFO] (__main__:285) - USING:: verbose = INFO [2024-11-30 07:01:55,970] [3310] [MainThread] [INFO] (__main__:285) - USING:: conf = [['models', 'localization_spine,localization_vertebra,segmentation_vertebra']] [2024-11-30 07:01:55,970] [3310] [MainThread] [INFO] (__main__:285) - USING:: host = 0.0.0.0 [2024-11-30 07:01:55,970] [3310] [MainThread] [INFO] (__main__:285) - USING:: port = 8000 [2024-11-30 07:01:55,970] [3310] [MainThread] [INFO] (__main__:285) - USING:: uvicorn_app = monailabel.app:app [2024-11-30 07:01:55,970] [3310] [MainThread] [INFO] (__main__:285) - USING:: ssl_keyfile = None [2024-11-30 07:01:55,970] [3310] [MainThread] [INFO] (__main__:285) - USING:: ssl_certfile = None [2024-11-30 07:01:55,970] [3310] [MainThread] [INFO] (__main__:285) - USING:: ssl_keyfile_password = None [2024-11-30 07:01:55,970] [3310] [MainThread] [INFO] (__main__:285) - USING:: ssl_ca_certs = None [2024-11-30 07:01:55,970] [3310] [MainThread] [INFO] (__main__:285) - USING:: workers = None [2024-11-30 07:01:55,970] [3310] [MainThread] [INFO] (__main__:285) - USING:: limit_concurrency = None [2024-11-30 07:01:55,970] [3310] [MainThread] [INFO] (__main__:285) - USING:: access_log = False [2024-11-30 07:01:55,970] [3310] [MainThread] [INFO] (__main__:285) - USING:: root_path = [2024-11-30 07:01:55,970] [3310] [MainThread] [INFO] (__main__:285) - USING:: log_level = info [2024-11-30 07:01:55,970] [3310] [MainThread] [INFO] (__main__:285) - USING:: log_config = None [2024-11-30 07:01:55,970] [3310] [MainThread] [INFO] (__main__:285) - USING:: dryrun = False [2024-11-30 07:01:55,970] [3310] [MainThread] [INFO] (__main__:285) - USING:: action = start_server [2024-11-30 07:01:55,971] [3310] [MainThread] [INFO] (__main__:296) - Allow Origins: ['*'] [2024-11-30 07:01:56,705] [3310] [MainThread] [INFO] (uvicorn.error:82) - Started server process [3310] [2024-11-30 07:01:56,705] [3310] [MainThread] [INFO] (uvicorn.error:48) - Waiting for application startup. App Init... [2024-11-30 07:01:56,705] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.app:37) - Initializing App from: /home/jun/iota/0-train_cleaned/LUNGND/apps/radiology; studies: /home/jun/iota/0-train_cleaned/LUNGND/datasets/spine; conf: {'models': 'localization_spine,localization_vertebra,segmentation_vertebra'} [2024-11-30 07:01:56,999] [3310] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for MONAILabelApp Found: [2024-11-30 07:01:57,000] [3310] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: [2024-11-30 07:01:57,001] [3310] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: [2024-11-30 07:01:57,001] [3310] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: [2024-11-30 07:01:57,002] [3310] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: [2024-11-30 07:01:57,002] [3310] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: [2024-11-30 07:01:57,002] [3310] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: [2024-11-30 07:01:57,003] [3310] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: [2024-11-30 07:01:57,003] [3310] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: [2024-11-30 07:01:57,003] [3310] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: [2024-11-30 07:01:57,003] [3310] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: [2024-11-30 07:01:57,003] [3310] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: [2024-11-30 07:01:57,004] [3310] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: [2024-11-30 07:01:57,004] [3310] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: [2024-11-30 07:01:57,004] [3310] [MainThread] [INFO] (main:101) - +++ Adding Model: localization_spine => lib.configs.localization_spine.LocalizationSpine [2024-11-30 07:01:57,102] [3310] [MainThread] [INFO] (main:101) - +++ Adding Model: localization_vertebra => lib.configs.localization_vertebra.LocalizationVertebra [2024-11-30 07:01:57,206] [3310] [MainThread] [INFO] (main:101) - +++ Adding Model: segmentation_vertebra => lib.configs.segmentation_vertebra.SegmentationVertebra [2024-11-30 07:01:57,307] [3310] [MainThread] [INFO] (main:104) - +++ Using Models: ['localization_spine', 'localization_vertebra', 'segmentation_vertebra'] [2024-11-30 07:01:57,307] [3310] [MainThread] [INFO] (monailabel.interfaces.app:135) - Init Datastore for: /home/jun/iota/0-train_cleaned/LUNGND/datasets/spine [2024-11-30 07:01:57,307] [3310] [MainThread] [INFO] (monailabel.datastore.local:130) - Auto Reload: True; Extensions: ['*.nii.gz', '*.nii', '*.nrrd', '*.jpg', '*.png', '*.tif', '*.svs', '*.xml'] [2024-11-30 07:01:57,309] [3310] [MainThread] [INFO] (monailabel.datastore.local:577) - Invalidate count: 0 [2024-11-30 07:01:57,309] [3310] [MainThread] [INFO] (monailabel.datastore.local:151) - Start observing external modifications on datastore (AUTO RELOAD) [2024-11-30 07:01:57,376] [3310] [MainThread] [INFO] (main:134) - +++ Adding Inferer:: localization_spine => [2024-11-30 07:01:57,376] [3310] [MainThread] [INFO] (main:134) - +++ Adding Inferer:: localization_vertebra => [2024-11-30 07:01:57,376] [3310] [MainThread] [INFO] (main:134) - +++ Adding Inferer:: segmentation_vertebra => [2024-11-30 07:01:57,376] [3310] [MainThread] [INFO] (main:220) - {'localization_spine': , 'localization_vertebra': , 'segmentation_vertebra': , 'Histogram+GraphCut': , 'GMM+GraphCut': , 'vertebra_pipeline': } [2024-11-30 07:01:57,376] [3310] [MainThread] [INFO] (main:235) - +++ Adding Trainer:: localization_spine => [2024-11-30 07:01:57,377] [3310] [MainThread] [INFO] (main:235) - +++ Adding Trainer:: localization_vertebra => [2024-11-30 07:01:57,377] [3310] [MainThread] [INFO] (main:235) - +++ Adding Trainer:: segmentation_vertebra => [2024-11-30 07:01:57,377] [3310] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: /home/jun/.cache/monailabel/sessions [2024-11-30 07:01:57,377] [3310] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600 [2024-11-30 07:01:57,377] [3310] [MainThread] [INFO] (monailabel.interfaces.app:469) - App Init - completed [2024-11-30 07:01:57,377] [timeloop] [INFO] Starting Timeloop.. [2024-11-30 07:01:57,377] [3310] [MainThread] [INFO] (timeloop:60) - Starting Timeloop.. [2024-11-30 07:01:57,378] [timeloop] [INFO] Registered job .run_scheduler at 0x7f02591d4280> [2024-11-30 07:01:57,378] [3310] [MainThread] [INFO] (timeloop:42) - Registered job .run_scheduler at 0x7f02591d4280> [2024-11-30 07:01:57,378] [timeloop] [INFO] Timeloop now started. Jobs will run based on the interval set [2024-11-30 07:01:57,378] [3310] [MainThread] [INFO] (timeloop:63) - Timeloop now started. Jobs will run based on the interval set [2024-11-30 07:01:57,378] [3310] [MainThread] [INFO] (uvicorn.error:62) - Application startup complete. [2024-11-30 07:01:57,378] [3310] [MainThread] [INFO] (uvicorn.error:214) - Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit) [2024-11-30 07:02:05,495] [3310] [MainThread] [INFO] (monailabel.endpoints.activelearning:44) - Active Learning Request: {'strategy': 'random', 'client_id': 'user-xyz'} [2024-11-30 07:02:05,495] [3310] [MainThread] [INFO] (monailabel.tasks.activelearning.random:47) - Random: Selected Image: verse004; Weight: 347 [2024-11-30 07:02:05,498] [3310] [MainThread] [INFO] (monailabel.endpoints.activelearning:60) - Next sample: {'id': 'verse004', 'weight': 347, 'path': '/home/jun/iota/0-train_cleaned/LUNGND/datasets/spine/verse004.nii.gz', 'ts': 1732920424, 'name': 'verse004.nii.gz', 'strategy': {'random': {'ts': 1732921325, 'client_id': 'user-xyz'}}} [2024-11-30 07:02:34,145] [3310] [MainThread] [INFO] (monailabel.endpoints.infer:171) - Infer Request: {'model': 'localization_spine', 'image': 'verse004', 'device': 'NVIDIA GeForce RTX 3070', 'result_extension': '.nrrd', 'result_dtype': 'uint8', 'client_id': 'user-xyz'} [2024-11-30 07:02:34,145] [3310] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:280) - Infer Request (final): {'device': 'cuda', 'model': 'localization_spine', 'image': '/home/jun/iota/0-train_cleaned/LUNGND/datasets/spine/verse004.nii.gz', 'result_extension': '.nrrd', 'result_dtype': 'uint8', 'client_id': 'user-xyz', 'description': 'A pre-trained model for volumetric (3D) spine localization from CT image'} [2024-11-30 07:02:34,146] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - PRE - Run Transform(s) [2024-11-30 07:02:34,146] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - PRE - Input Keys: ['device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path'] [2024-11-30 07:02:35,892] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (LoadImaged): Time: 1.7449; image: torch.Size([161, 338, 61])(torch.float32) [2024-11-30 07:02:36,557] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (EnsureTyped): Time: 0.6655; image: torch.Size([161, 338, 61])(torch.float32) [2024-11-30 07:02:36,558] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (EnsureChannelFirstd): Time: 0.0002; image: torch.Size([1, 161, 338, 61])(torch.float32) [2024-11-30 07:02:36,559] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (CacheObjectd): Time: 0.0015; image: torch.Size([1, 161, 338, 61])(torch.float32) [2024-11-30 07:02:37,304] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (Spacingd): Time: 0.7443; image: torch.Size([1, 124, 260, 93])(torch.float32) [2024-11-30 07:02:37,305] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (ScaleIntensityRanged): Time: 0.0007; image: torch.Size([1, 124, 260, 93])(torch.float32) [2024-11-30 07:02:37,762] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (GaussianSmoothd): Time: 0.4574; image: torch.Size([1, 124, 260, 93])(torch.float32) [2024-11-30 07:02:37,763] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (ScaleIntensityd): Time: 0.0007; image: torch.Size([1, 124, 260, 93])(torch.float32) [2024-11-30 07:02:37,764] [3310] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:504) - Inferer:: cuda => SlidingWindowInferer => {'roi_size': (96, 96, 96), 'sw_batch_size': 2, 'overlap': 0.4, 'mode': gaussian, 'sigma_scale': 0.125, 'padding_mode': 'replicate', 'cval': 0.0, 'sw_device': None, 'device': None, 'progress': False, 'cpu_thresh': None, 'buffer_steps': None, 'buffer_dim': -1, 'with_coord': False, 'roi_weight_map': None} [2024-11-30 07:02:37,764] [3310] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:432) - Infer model path: /home/jun/iota/0-train_cleaned/LUNGND/apps/radiology/model/pretrained_localization_spine.pt [2024-11-30 07:02:38,465] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - POST - Run Transform(s) [2024-11-30 07:02:38,465] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - POST - Input Keys: ['device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path', 'latencies', 'image_cached', 'pred'] [2024-11-30 07:02:38,466] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (EnsureTyped): Time: 0.0001; image: torch.Size([1, 124, 260, 93])(torch.float32); pred: torch.Size([25, 124, 260, 93])(torch.float32) [2024-11-30 07:02:38,472] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (Activationsd): Time: 0.0059; image: torch.Size([1, 124, 260, 93])(torch.float32); pred: torch.Size([25, 124, 260, 93])(torch.float32) [2024-11-30 07:02:38,472] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (AsDiscreted): Time: 0.0004; image: torch.Size([1, 124, 260, 93])(torch.float32); pred: torch.Size([1, 124, 260, 93])(torch.float32) [2024-11-30 07:02:38,489] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (KeepLargestConnectedComponentd): Time: 0.0168; image: torch.Size([1, 124, 260, 93])(torch.float32); pred: torch.Size([1, 124, 260, 93])(torch.float32) [2024-11-30 07:02:38,490] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (BinaryMaskd): Time: 0.0003; image: torch.Size([1, 124, 260, 93])(torch.float32); pred: torch.Size([1, 124, 260, 93])(torch.float32) [2024-11-30 07:02:38,491] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (Restored): Time: 0.001; image: torch.Size([1, 124, 260, 93])(torch.float32); pred: torch.Size([161, 338, 61])(torch.float32) [2024-11-30 07:02:38,491] [3310] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:604) - Writing Result... [2024-11-30 07:02:38,491] [3310] [MainThread] [INFO] (monailabel.transform.writer:196) - Result ext: .nrrd; write_to_file: True; dtype: uint8 [2024-11-30 07:02:38,508] [3310] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:335) - ++ Latencies => Total: 4.3631; Pre: 3.6183; Inferer: 0.7010; Invert: 0.0000; Post: 0.0265; Write: 0.0169 [2024-11-30 07:02:38,508] [3310] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:370) - Result File: /tmp/tmpk6mosra4.nrrd [2024-11-30 07:02:38,508] [3310] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:371) - Result Json Keys: ['label_names', 'latencies', 'centroids'] [2024-11-30 07:02:59,822] [3310] [MainThread] [INFO] (monailabel.endpoints.infer:171) - Infer Request: {'model': 'localization_vertebra', 'image': 'verse004', 'device': 'NVIDIA GeForce RTX 3070', 'result_extension': '.nrrd', 'result_dtype': 'uint8', 'client_id': 'user-xyz'} [2024-11-30 07:02:59,824] [3310] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:280) - Infer Request (final): {'device': 'cuda', 'model': 'localization_vertebra', 'image': '/home/jun/iota/0-train_cleaned/LUNGND/datasets/spine/verse004.nii.gz', 'result_extension': '.nrrd', 'result_dtype': 'uint8', 'client_id': 'user-xyz', 'description': 'A pre-trained model for volumetric (3D) vertebra localization from CT image'} [2024-11-30 07:02:59,825] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - PRE - Run Transform(s) [2024-11-30 07:02:59,825] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - PRE - Input Keys: ['device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path'] [2024-11-30 07:02:59,890] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (LoadImaged): Time: 0.0645; image: torch.Size([161, 338, 61])(torch.float32) [2024-11-30 07:02:59,895] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (EnsureTyped): Time: 0.0051; image: torch.Size([161, 338, 61])(torch.float32) [2024-11-30 07:02:59,896] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (EnsureChannelFirstd): Time: 0.0002; image: torch.Size([1, 161, 338, 61])(torch.float32) [2024-11-30 07:02:59,897] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (CacheObjectd): Time: 0.0009; image: torch.Size([1, 161, 338, 61])(torch.float32) [2024-11-30 07:02:59,929] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (Spacingd): Time: 0.0324; image: torch.Size([1, 124, 260, 93])(torch.float32) [2024-11-30 07:02:59,930] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (ScaleIntensityRanged): Time: 0.0005; image: torch.Size([1, 124, 260, 93])(torch.float32) [2024-11-30 07:03:00,131] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (GaussianSmoothd): Time: 0.2011; image: torch.Size([1, 124, 260, 93])(torch.float32) [2024-11-30 07:03:00,149] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (ScaleIntensityd): Time: 0.0169; image: torch.Size([1, 124, 260, 93])(torch.float32) [2024-11-30 07:03:00,149] [3310] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:504) - Inferer:: cuda => SlidingWindowInferer => {'roi_size': (96, 96, 96), 'sw_batch_size': 2, 'overlap': 0.4, 'mode': gaussian, 'sigma_scale': 0.125, 'padding_mode': 'replicate', 'cval': 0.0, 'sw_device': None, 'device': device(type='cpu'), 'progress': False, 'cpu_thresh': None, 'buffer_steps': None, 'buffer_dim': -1, 'with_coord': False, 'roi_weight_map': None} [2024-11-30 07:03:00,149] [3310] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:432) - Infer model path: /home/jun/iota/0-train_cleaned/LUNGND/apps/radiology/model/pretrained_localization_vertebra.pt [2024-11-30 07:03:01,073] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - INV - Run Transform(s) [2024-11-30 07:03:01,073] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - INV - Input Keys: ['device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path', 'latencies', 'image_cached', 'pred'] [2024-11-30 07:03:03,072] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - INV - Transform (Spacingd): Time: 1.9987; image: torch.Size([25, 161, 338, 61])(torch.float32); pred: torch.Size([25, 124, 260, 93])(torch.float32) [2024-11-30 07:03:03,075] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - POST - Run Transform(s) [2024-11-30 07:03:03,075] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - POST - Input Keys: ['device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path', 'latencies', 'image_cached', 'pred'] [2024-11-30 07:03:03,076] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (EnsureTyped): Time: 0.0001; image: torch.Size([1, 124, 260, 93])(torch.float32); pred: torch.Size([25, 161, 338, 61])(torch.float32) [2024-11-30 07:03:03,163] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (Activationsd): Time: 0.0875; image: torch.Size([1, 124, 260, 93])(torch.float32); pred: torch.Size([25, 161, 338, 61])(torch.float32) [2024-11-30 07:03:03,455] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (AsDiscreted): Time: 0.2918; image: torch.Size([1, 124, 260, 93])(torch.float32); pred: torch.Size([1, 161, 338, 61])(torch.float32) [2024-11-30 07:03:03,476] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (KeepLargestConnectedComponentd): Time: 0.0201; image: torch.Size([1, 124, 260, 93])(torch.float32); pred: torch.Size([1, 161, 338, 61])(torch.float32) [2024-11-30 07:03:03,476] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (Restored): Time: 0.0002; image: torch.Size([1, 124, 260, 93])(torch.float32); pred: torch.Size([161, 338, 61])(torch.float32) [2024-11-30 07:03:03,496] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (VertebraLocalizationSegmentation): Time: 0.0191; image: torch.Size([1, 124, 260, 93])(torch.float32); pred: torch.Size([161, 338, 61])(torch.float32) [2024-11-30 07:03:03,497] [3310] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:604) - Writing Result... [2024-11-30 07:03:03,497] [3310] [MainThread] [INFO] (monailabel.transform.writer:196) - Result ext: .nrrd; write_to_file: True; dtype: uint8 [2024-11-30 07:03:03,514] [3310] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:335) - ++ Latencies => Total: 3.6908; Pre: 0.3249; Inferer: 0.8780; Invert: 2.0473; Post: 0.4225; Write: 0.0170 [2024-11-30 07:03:03,514] [3310] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:370) - Result File: /tmp/tmpqbk2jkp5.nrrd [2024-11-30 07:03:03,514] [3310] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:371) - Result Json Keys: ['centroids', 'label_names', 'latencies'] [2024-11-30 07:03:09,494] [3310] [MainThread] [INFO] (monailabel.endpoints.infer:171) - Infer Request: {'model': 'vertebra_pipeline', 'image': 'verse004', 'device': 'NVIDIA GeForce RTX 3070', 'result_extension': '.nrrd', 'result_dtype': 'uint8', 'client_id': 'user-xyz'} [2024-11-30 07:03:09,494] [3310] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:280) - Infer Request (final): {'device': 'cuda', 'model': 'vertebra_pipeline', 'image': '/home/jun/iota/0-train_cleaned/LUNGND/datasets/spine/verse004.nii.gz', 'result_extension': '.nrrd', 'result_dtype': 'uint8', 'client_id': 'user-xyz', 'description': 'Combines three stage for vertebra segmentation', 'image_path': '/home/jun/iota/0-train_cleaned/LUNGND/datasets/spine/verse004.nii.gz', 'pipeline_mode': True} [2024-11-30 07:03:09,495] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - PRE - Run Transform(s) [2024-11-30 07:03:09,495] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - PRE - Input Keys: ['device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path', 'pipeline_mode'] [2024-11-30 07:03:09,556] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (LoadImaged): Time: 0.0605; image: torch.Size([161, 338, 61])(torch.float32) [2024-11-30 07:03:09,561] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (EnsureTyped): Time: 0.0049; image: torch.Size([161, 338, 61])(torch.float32) [2024-11-30 07:03:09,561] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (EnsureChannelFirstd): Time: 0.0001; image: torch.Size([1, 161, 338, 61])(torch.float32) [2024-11-30 07:03:09,562] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (CacheObjectd): Time: 0.0005; image: torch.Size([1, 161, 338, 61])(torch.float32) [2024-11-30 07:03:09,809] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (Spacingd): Time: 0.2472; image: torch.Size([1, 124, 260, 93])(torch.float32) [2024-11-30 07:03:09,810] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (ScaleIntensityRanged): Time: 0.0004; image: torch.Size([1, 124, 260, 93])(torch.float32) [2024-11-30 07:03:09,829] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (GaussianSmoothd): Time: 0.0188; image: torch.Size([1, 124, 260, 93])(torch.float32) [2024-11-30 07:03:09,830] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (ScaleIntensityd): Time: 0.0007; image: torch.Size([1, 124, 260, 93])(torch.float32) [2024-11-30 07:03:09,832] [3310] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:504) - Inferer:: cuda => SlidingWindowInferer => {'roi_size': (96, 96, 96), 'sw_batch_size': 2, 'overlap': 0.4, 'mode': gaussian, 'sigma_scale': 0.125, 'padding_mode': 'replicate', 'cval': 0.0, 'sw_device': None, 'device': None, 'progress': False, 'cpu_thresh': None, 'buffer_steps': None, 'buffer_dim': -1, 'with_coord': False, 'roi_weight_map': None} [2024-11-30 07:03:09,832] [3310] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:432) - Infer model path: /home/jun/iota/0-train_cleaned/LUNGND/apps/radiology/model/pretrained_localization_spine.pt [2024-11-30 07:03:10,373] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - POST - Run Transform(s) [2024-11-30 07:03:10,374] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - POST - Input Keys: ['device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path', 'pipeline_mode', 'latencies', 'image_cached', 'pred'] [2024-11-30 07:03:10,374] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (EnsureTyped): Time: 0.0001; image: torch.Size([1, 124, 260, 93])(torch.float32); pred: torch.Size([25, 124, 260, 93])(torch.float32) [2024-11-30 07:03:10,380] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (Activationsd): Time: 0.006; image: torch.Size([1, 124, 260, 93])(torch.float32); pred: torch.Size([25, 124, 260, 93])(torch.float32) [2024-11-30 07:03:10,381] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (AsDiscreted): Time: 0.0002; image: torch.Size([1, 124, 260, 93])(torch.float32); pred: torch.Size([1, 124, 260, 93])(torch.float32) [2024-11-30 07:03:10,396] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (KeepLargestConnectedComponentd): Time: 0.0155; image: torch.Size([1, 124, 260, 93])(torch.float32); pred: torch.Size([1, 124, 260, 93])(torch.float32) [2024-11-30 07:03:10,397] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (BinaryMaskd): Time: 0.0002; image: torch.Size([1, 124, 260, 93])(torch.float32); pred: torch.Size([1, 124, 260, 93])(torch.float32) [2024-11-30 07:03:10,398] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (Restored): Time: 0.0008; image: torch.Size([1, 124, 260, 93])(torch.float32); pred: torch.Size([161, 338, 61])(torch.float32) [2024-11-30 07:03:10,398] [3310] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:335) - ++ Latencies => Total: 0.9038; Pre: 0.3372; Inferer: 0.5414; Invert: 0.0000; Post: 0.0248; Write: 0.0000 [2024-11-30 07:03:10,398] [3310] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:371) - Result Json Keys: ['label_names', 'latencies', 'centroids'] [2024-11-30 07:03:10,398] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:24) - **************************** DATA ******************************************** [2024-11-30 07:03:10,398] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: device = cuda [2024-11-30 07:03:10,398] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: model = vertebra_pipeline [2024-11-30 07:03:10,399] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: image = torch.Size([1, 161, 338, 61]) [2024-11-30 07:03:10,399] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: result_extension = .nrrd [2024-11-30 07:03:10,399] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: result_dtype = uint8 [2024-11-30 07:03:10,399] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: client_id = user-xyz [2024-11-30 07:03:10,399] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: description = Combines three stage for vertebra segmentation [2024-11-30 07:03:10,399] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: image_path = /home/jun/iota/0-train_cleaned/LUNGND/datasets/spine/verse004.nii.gz [2024-11-30 07:03:10,399] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: label = torch.Size([161, 338, 61]) [2024-11-30 07:03:10,399] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: pipeline_mode = True [2024-11-30 07:03:10,399] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:38) - ****************************************************************************** monai.transforms.croppad.dictionary CropForegroundd.__init__:allow_smaller: Current default value of argument `allow_smaller=True` has been deprecated since version 1.2. It will be changed to `allow_smaller=False` in version 1.5. [2024-11-30 07:03:10,400] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - PRE - Run Transform(s) [2024-11-30 07:03:10,400] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - PRE - Input Keys: ['device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path', 'label', 'pipeline_mode'] [2024-11-30 07:03:10,401] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (EnsureChannelFirstd): Time: 0.0002; image: torch.Size([1, 161, 338, 61])(torch.float32); label: torch.Size([1, 161, 338, 61])(torch.float32) [2024-11-30 07:03:10,401] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (CacheObjectd): Time: 0.0005; image: torch.Size([1, 161, 338, 61])(torch.float32); label: torch.Size([1, 161, 338, 61])(torch.float32) [2024-11-30 07:03:10,423] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (Spacingd): Time: 0.0216; image: torch.Size([1, 124, 260, 93])(torch.float32); label: torch.Size([1, 124, 260, 92])(torch.float32) [2024-11-30 07:03:10,424] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (ScaleIntensityRanged): Time: 0.0004; image: torch.Size([1, 124, 260, 93])(torch.float32); label: torch.Size([1, 124, 260, 92])(torch.float32) [2024-11-30 07:03:10,440] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (GaussianSmoothd): Time: 0.0166; image: torch.Size([1, 124, 260, 93])(torch.float32); label: torch.Size([1, 124, 260, 92])(torch.float32) [2024-11-30 07:03:10,441] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (ScaleIntensityd): Time: 0.0007; image: torch.Size([1, 124, 260, 93])(torch.float32); label: torch.Size([1, 124, 260, 92])(torch.float32) [2024-11-30 07:03:10,444] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (CropForegroundd): Time: 0.0028; image: torch.Size([1, 0, 0, 0])(torch.float32); label: torch.Size([1, 0, 0, 0])(torch.float32) [2024-11-30 07:03:10,445] [3310] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:504) - Inferer:: cuda => SlidingWindowInferer => {'roi_size': (96, 96, 96), 'sw_batch_size': 2, 'overlap': 0.4, 'mode': gaussian, 'sigma_scale': 0.125, 'padding_mode': 'replicate', 'cval': 0.0, 'sw_device': None, 'device': device(type='cpu'), 'progress': False, 'cpu_thresh': None, 'buffer_steps': None, 'buffer_dim': -1, 'with_coord': False, 'roi_weight_map': None} [2024-11-30 07:03:10,445] [3310] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:432) - Infer model path: /home/jun/iota/0-train_cleaned/LUNGND/apps/radiology/model/pretrained_localization_vertebra.pt [2024-11-30 07:03:10,453] [3310] [MainThread] [ERROR] (uvicorn.error:412) - Exception in ASGI application Traceback (most recent call last): File "/home/jun/anaconda3/envs/lungmonai/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi result = await app( # type: ignore[func-returns-value] File "/home/jun/anaconda3/envs/lungmonai/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__ return await self.app(scope, receive, send) File "/home/jun/anaconda3/envs/lungmonai/lib/python3.8/site-packages/fastapi/applications.py", line 1054, in __call__ await super().__call__(scope, receive, send) File "/home/jun/anaconda3/envs/lungmonai/lib/python3.8/site-packages/starlette/applications.py", line 123, in __call__ await self.middleware_stack(scope, receive, send) File "/home/jun/anaconda3/envs/lungmonai/lib/python3.8/site-packages/starlette/middleware/errors.py", line 186, in __call__ raise exc File "/home/jun/anaconda3/envs/lungmonai/lib/python3.8/site-packages/starlette/middleware/errors.py", line 164, in __call__ await self.app(scope, receive, _send) File "/home/jun/anaconda3/envs/lungmonai/lib/python3.8/site-packages/starlette/middleware/cors.py", line 85, in __call__ await self.app(scope, receive, send) File "/home/jun/anaconda3/envs/lungmonai/lib/python3.8/site-packages/starlette/middleware/exceptions.py", line 65, in __call__ await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "/home/jun/anaconda3/envs/lungmonai/lib/python3.8/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "/home/jun/anaconda3/envs/lungmonai/lib/python3.8/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "/home/jun/anaconda3/envs/lungmonai/lib/python3.8/site-packages/starlette/routing.py", line 756, in __call__ await self.middleware_stack(scope, receive, send) File "/home/jun/anaconda3/envs/lungmonai/lib/python3.8/site-packages/starlette/routing.py", line 776, in app await route.handle(scope, receive, send) File "/home/jun/anaconda3/envs/lungmonai/lib/python3.8/site-packages/starlette/routing.py", line 297, in handle await self.app(scope, receive, send) File "/home/jun/anaconda3/envs/lungmonai/lib/python3.8/site-packages/starlette/routing.py", line 77, in app await wrap_app_handling_exceptions(app, request)(scope, receive, send) File "/home/jun/anaconda3/envs/lungmonai/lib/python3.8/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "/home/jun/anaconda3/envs/lungmonai/lib/python3.8/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "/home/jun/anaconda3/envs/lungmonai/lib/python3.8/site-packages/starlette/routing.py", line 72, in app response = await func(request) File "/home/jun/anaconda3/envs/lungmonai/lib/python3.8/site-packages/fastapi/routing.py", line 278, in app raw_response = await run_endpoint_function( File "/home/jun/anaconda3/envs/lungmonai/lib/python3.8/site-packages/fastapi/routing.py", line 191, in run_endpoint_function return await dependant.call(**values) File "/home/jun/anaconda3/envs/lungmonai/lib/python3.8/site-packages/monailabel/endpoints/infer.py", line 206, in api_run_inference return run_inference(background_tasks, model, image, session_id, params, file, label, output) File "/home/jun/anaconda3/envs/lungmonai/lib/python3.8/site-packages/monailabel/endpoints/infer.py", line 172, in run_inference result = instance.infer(request) File "/home/jun/anaconda3/envs/lungmonai/lib/python3.8/site-packages/monailabel/interfaces/app.py", line 307, in infer result_file_name, result_json = task(request) File "/home/jun/iota/0-train_cleaned/LUNGND/apps/radiology/lib/infers/vertebra_pipeline.py", line 149, in __call__ d2, r2, l2 = self.locate_vertebra(request, image, label) File "/home/jun/iota/0-train_cleaned/LUNGND/apps/radiology/lib/infers/vertebra_pipeline.py", line 82, in locate_vertebra d, r = self.task_loc_vertebra(req) File "/home/jun/anaconda3/envs/lungmonai/lib/python3.8/site-packages/monailabel/tasks/infer/basic_infer.py", line 307, in __call__ data = self.run_inferer(data, device=device) File "/home/jun/anaconda3/envs/lungmonai/lib/python3.8/site-packages/monailabel/tasks/infer/basic_infer.py", line 514, in run_inferer outputs = inferer(inputs, network) File "/home/jun/anaconda3/envs/lungmonai/lib/python3.8/site-packages/monai/inferers/inferer.py", line 515, in __call__ return sliding_window_inference( File "/home/jun/anaconda3/envs/lungmonai/lib/python3.8/site-packages/monai/inferers/utils.py", line 171, in sliding_window_inference inputs = F.pad(inputs, pad=pad_size, mode=look_up_option(padding_mode, PytorchPadMode), value=cval) RuntimeError: Expected 4D or 5D (batch mode) tensor with possibly 0 batch size and other non-zero dimensions for input, but got: [1, 1, 0, 0, 0]