abnormal training time when run multi NeuRad jobs

[szhang963]

The training time becomes longer when I run the second job in a multi-GPU cluster.

And then, the second job's training time is also slower as below.

Could you give me some suggestions?
Thank you in advance.

[georghess]

Have you checked that the jobs do not use the same resources (GPU, CPU)?

[szhang963]

I can ensure I do not use the same GPU, but for the CPU I am not sure.
The training time is not stable when running multiple jobs in a multi-GPU cluster.

I did not meet the problem in the nerfstudio project.

Can you reproduce the issue?

[georghess]

I see. We often train multiple jobs in parallel on our cluster as well and have never had any issues where they affect each other. I know that the multiprocess data loading has been given some people issues, not sure if that is the case here as well?
You could try running the training with --pipeline.datamanager.num_processes=0 and see if that helps. Do you see the GPU-utilization dropping when running multiple jobs?

[szhang963]

Hi, the issue was solved by setting --pipeline.datamanager.num_processes=0. At the same time, the parameter does not affect the training time in a single job for me. So, that is why? How much can it speed up for you?
Thank you for your help.