Inconsistent output of logger message in NNunet train stage

[iffthomas]

Describe the bug
logger output during training can deviate when using custom made splits.

Please only create issues here relating to the functioning of the tutorials. For all other questions/problems/bugs, please refer to MONAI's main repository. For MONAI bugs (not just a problem in the notebook), please create an issue. For questions, please use the Discussions tab.

To Reproduce
Steps to reproduce the behavior:

• have already a splits_final.json file in the correct format
• run training it with nnUNetTrainerv2 train
• if it doesn't give out the correct logger input rerun
  Expected behavior
  2023-11-16 09:58:49.044835: unpacking dataset...
  2023-11-16 09:59:21.353696: unpacking done...
  2023-11-16 09:59:21.355801: do_dummy_2d_data_aug: False
  2023-11-16 09:59:21.365563: Creating new 5-fold cross-validation split...
  2023-11-16 09:59:21.369237: Desired fold for training: 0
  2023-11-16 09:59:21.369696: This split has 156 training and 40 validation cases.
  2023-11-16 09:59:21.460418: Unable to plot network architecture:
  2023-11-16 09:59:21.460952: No module named 'hiddenlayer'
  2023-11-16 09:59:21.480445:
  2023-11-16 09:59:21.480848: Epoch 0
  2023-11-16 09:59:21.481394: Current learning rate: 0.01

this behavior happened after just rerunning it after I canceled the job
2023-11-16 10:08:15.432898: unpacking dataset...
2023-11-16 10:08:16.039125: unpacking done...
2023-11-16 10:08:16.041371: do_dummy_2d_data_aug: False
2023-11-16 10:08:16.043030: Using splits from existing split file: ./data/nnUNet_preprocessed/Dataset006_SPIDER/splits_final.json
2023-11-16 10:08:16.044067: The split file contains 5 splits.
2023-11-16 10:08:16.044394: Desired fold for training: 0
2023-11-16 10:08:16.044732: This split has 156 training and 40 validation cases.
2023-11-16 10:08:16.106095: Unable to plot network architecture:
2023-11-16 10:08:16.106503: No module named 'hiddenlayer'
2023-11-16 10:08:16.120372:
2023-11-16 10:08:16.120819: Epoch 0
2023-11-16 10:08:16.121274: Current learning rate: 0.01

Important note : the splits that get created are still the same i guess, I have slightly imbalanced splits and the number is still the same which would be an indication that just the logger message is faulty.

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• OS
• Python version
• MONAI version [e.g. git commit hash]
• CUDA/cuDNN version
• GPU models and configuration

Additional context
Add any other context about the problem here.

[KumoLiu]

Hi @iffthomas, I didn't get the point what inconsistent logger info do you refer to?

[iffthomas]

Hi @KumoLiu
The logger message highlighted in bold is different while not changeing the splits.json file or changing any command for the same dataset.

If I run only train for the first time after I've ran plan_and_preprocess with the splits.json file i get this message in the logger:
2023-11-16 09:59:21.365563: Creating new 5-fold cross-validation split...

While this is wrong I cancel the job and resubmit it so: i get this message in the logger:
2023-11-16 10:08:16.043030: Using splits from existing split file: ./data/nnUNet_preprocessed/Dataset006_SPIDER/splits_final.json

[KumoLiu]

Hi @iffthomas, I think that's expected unless you remove the "splits_final.json" otherwise it may use the splits from the existing split file.
Find more information here. https://github.com/MIC-DKFZ/nnUNet/blob/master/documentation/manual_data_splits.md

[iffthomas]

Hi KumoLiu,

I see where the problem is: the splits from ma nnUNetraw folder and nnUNetpreprocessed aren't the same. the reason for this is that my splits don't get transfered when doing plan_and_preprocess from the raw folder to the preprocessed folder

I will have a look wheter it's possible for nnUnet to transfer this file during the plan_and_preprocess phase

[KumoLiu]

Convert to discussion for now, feel free to create another one if meet any issues. Thanks!