Optimal number of workers

[eberrigan]

Hey Abe and community,

I was working with some pretty high resolution images so I cut them into lots of tiles (12 900px tiles per image). I got an error I had not seen before when running training locally on my gaming laptop.

"ImportError: DLL load failed while importing _ckdtree: The paging file is too small for this operation to complete."

After doing some googling, I found that other people had similar issues with uploading data (in various softwares) due to limited processing power and it was fixed by decreasing the number of workers. I decreased the number of maxworkers from 12 to 6 here:

root_painter/trainer/main.py

Line 39 in 78eda63

trainer = Trainer(sync_dir=args.syncdir, patch_size=args.patchsize, max_workers=args.maxworkers)

And that fixed the problem. I am not to sure of how this part of your software works. Should it be able to parallelize uploading the data? Do you have a rule of thumb given the CPU of your computer for maxworkers? Is it able to use the GPU for data uploading?

Thanks so much for your help!

Elizabeth

[Abe404]

Hi Elizabeth,

Thanks for letting me know about this problem.

Although the default maxworkers is 12, this is just the max. The number of workers that will be used is also based on the available number of cores on your machine (detected by python). See:

root_painter/trainer/trainer.py

Line 80 in 78eda63

self.num_workers=min(multiprocessing.cpu_count(), max_workers)

I'm not yet sure what could have caused the specific 'DLL load failed while importing _ckdtree' error on your machine, so Im not yet sure what the best fix is. One option might be to try to automatically detect this error and automatically try again with less workers, but until we have a better understanding of this problem, for now I think your solution of just manually specifying the maxworkers on your laptop makes a lot of sense. You can also pass this as a command line option when starting the trainer, so you don't need to modify the code.

Kind regards,
Abraham

[eberrigan]

Thanks so much! That line of code makes a lot of sense and I can see why I haven't gotten this error before (I didn't realize it was counting the number of available cores on my machine). I will let you know if I get it again.