You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In #2269, specifically d0b3afa, I reduce the capacity of the multi-threading queue that stores the shuffling matrices, since there isn't really any need to buffer data there, it may use extra memory for large datasets, and has an added benefit of providing more faithful progress information since that's reported by the source.
I decided heuristically to set the capacity of the queue to the number of threads in use.
However I found that when using -nthreads 1, such that the threading queue is set up (unlike -nthreads 0) but has capacity 1, the program hangs.
Might not be worth the effort trying to get the code working robustly with a capacity of 1. Can just make sure it's at least 2. But will need to make sure it's done for both normal and ordered queues.
The text was updated successfully, but these errors were encountered:
In #2269, specifically d0b3afa, I reduce the capacity of the multi-threading queue that stores the shuffling matrices, since there isn't really any need to buffer data there, it may use extra memory for large datasets, and has an added benefit of providing more faithful progress information since that's reported by the source.
I decided heuristically to set the capacity of the queue to the number of threads in use.
However I found that when using
-nthreads 1
, such that the threading queue is set up (unlike-nthreads 0
) but has capacity 1, the program hangs.Might not be worth the effort trying to get the code working robustly with a capacity of 1. Can just make sure it's at least 2. But will need to make sure it's done for both normal and ordered queues.
The text was updated successfully, but these errors were encountered: