Training speed

[HarukiYqM]

Thanks for this nice work. Could you provide a rough estimation of the running time for this implementation?

Currently, it takes around 2.5 hours to train one epoch and seems much slower than the normal case. (Total batch size 2048, 4 x 8V100, 32G)

Thank you!

[HarukiYqM]

In addition, it seems that there is one node always idle during forward pass.

[antoine77340]

HI,

If I remember correctly, it was a bit faster to run this code, i.e. around one hour per epoch.
One key thing to make the training fast is to ensure a very fast IO because IO is a bottleneck.
Do you have the videos stored on SSD disk? We did have all videos stored in a distributed SSD disk and this was already input bound so I guess it could be even worse with HDD disk?

[HarukiYqM]

HI,

If I remember correctly, it was a bit faster to run this code, i.e. around one hour per epoch.
One key thing to make the training fast is to ensure a very fast IO because IO is a bottleneck.
Do you have the videos stored on SSD disk? We did have all videos stored in a distributed SSD disk and this was already input bound so I guess it could be even worse with HDD disk?

Thanks for your reply. It is indeed the IO slow down the training. Currently, I have to store the data on the cloud and cannot use local SSD. Do you have any suggestions to make the training possible with a cloud computing system like azure? Also I wonder that will the center crop option reduces the performance?