Very slow training for lung nodule CT detection

[Thibescobar]

Hello,

I am just getting into MONAI for a lung nodule detection project. The proposed model in the model zoo seems the go-to to me. I managed to launch a training using the bundle commands, but I find it relatively slow...

There are 507 iterations. For 300 epochs (default), it thus gives 300×507=152100 iterations in total.
My concern is that an iteration take approximately 30s. If we do the calculation it makes 152100×30=4563000s s=1268h=53d...

If someone has some help it would be great.
Thank you !

My configuration is:

• Computer: Laptop Dell Precision 7670

• OS: Windows 10 Professional (22H2)

• System type: x64

• GPU: NVIDIA RTX A3000 12GB

• CPU: 12th Gen Intel(R) Core(TM) i7-12850HX 2.10 GHz

• RAM: 32GB

• Python version: 3.10.14

• MONAI version: 1.3.0

• MONAI Weekly version: 1.4.dev2414

• Pytorch version: torch 2.2.2+cu118

• cuDNN version:
  - 8.7 given by conda activate monailuna && python >>> import torch >>> torch.backends.cudnn.version() (I think this is this one)
  - 8.6 installed outside the active conda env at C:\Program Files\NVIDIA GPU Computing Toolkit\CUDNN

• CUDA version:
  - 11.8 given by conda activate monailuna && python >>> import torch >>> torch.version.cuda (I think this is this one)
  - 11.7 given by nvcc --version (version installed outside the active conda env)
  - 12.2 given by nvidia-smi (compatible version but not the one installed?)

Terminal example:

(monailuna) C:\lung_nodule_ct_detection>python -m monai.bundle run --config_file configs/train.json
2024-04-02 17:40:55,997 - INFO - --- input summary of monai.bundle.scripts.run ---
2024-04-02 17:40:55,998 - INFO - > config_file: 'configs/train.json'
2024-04-02 17:40:55,999 - INFO - ---
C:\AppData\Local\anaconda3\envs\monailuna\lib\site-packages\monai\utils\deprecate_utils.py:321: FutureWarning: monai.bundle.workflows ConfigWorkflow.__init__:workflow_type: Current default value of argument `workflow_type=None` has been deprecated since version 1.2. It will be changed to `workflow_type=train` in version 1.4.
  warn_deprecated(argname, msg, warning_category)
2024-04-02 17:40:55,999 - INFO - Setting logging properties based on config: configs\logging.conf.
2024-04-02 17:40:56,787 - root - INFO - Running ATSS Matching with num_candidates=4 and center_in_gt False.
2024-04-02 17:40:56,787 - root - INFO - Sampling hard negatives on a per batch basis
2024-04-02 17:40:56,828 - ignite.engine.engine.DetectionTrainer - INFO - Engine run resuming from iteration 0, epoch 0 until 300 epochs
2024-04-02 17:41:57,733 - INFO - Epoch: 1/300, Iter: 1/507 -- train_loss: 0.9995
2024-04-02 17:42:25,909 - INFO - Epoch: 1/300, Iter: 2/507 -- train_loss: 1.6064
2024-04-02 17:42:54,161 - INFO - Epoch: 1/300, Iter: 3/507 -- train_loss: 1.0635
2024-04-02 17:43:22,382 - INFO - Epoch: 1/300, Iter: 4/507 -- train_loss: 1.3047
2024-04-02 17:43:49,455 - INFO - Epoch: 1/300, Iter: 5/507 -- train_loss: 1.1992
2024-04-02 17:44:16,477 - INFO - Epoch: 1/300, Iter: 6/507 -- train_loss: 1.0225
2024-04-02 17:44:43,447 - INFO - Epoch: 1/300, Iter: 7/507 -- train_loss: 1.2119
2024-04-02 17:45:10,548 - INFO - Epoch: 1/300, Iter: 8/507 -- train_loss: 0.7261
2024-04-02 17:45:37,602 - INFO - Epoch: 1/300, Iter: 9/507 -- train_loss: 1.4824
2024-04-02 17:46:04,573 - INFO - Epoch: 1/300, Iter: 10/507 -- train_loss: 1.2246
2024-04-02 17:46:31,649 - INFO - Epoch: 1/300, Iter: 11/507 -- train_loss: 0.8862
2024-04-02 17:46:58,708 - INFO - Epoch: 1/300, Iter: 12/507 -- train_loss: 1.3828
2024-04-02 17:47:25,722 - INFO - Epoch: 1/300, Iter: 13/507 -- train_loss: 1.0215
2024-04-02 17:47:52,613 - INFO - Epoch: 1/300, Iter: 14/507 -- train_loss: 1.1992
2024-04-02 17:48:19,598 - INFO - Epoch: 1/300, Iter: 15/507 -- train_loss: 1.3340
2024-04-02 17:48:46,623 - INFO - Epoch: 1/300, Iter: 16/507 -- train_loss: 1.4902
[...]

Edit:
Related to Project-MONAI/MONAI#7619