[question] Transfering work units from a pool to another

[mdorier]

Assuming I have a pool P1, that I know (1) is not used by any scheduler/xstream at the moment, (2) has no currently running ULT or task that could return to it upon yielding, and (3) will not be used as an argument to ABT_thread_create or other functions, i.e. ULTs that are in the pool are going to stay there and no new ULTs will be pushed or created in that pool.

I want to migrate the content of P1 to a pool P2 (which is in use by a scheduler/xstream, and has MPMC access). Is the following the right way of doing such a migration:

1. Use ABT_pool_get_size to get the number of ULTs in P1
2. Use ABT_pool_pop_threads to pop them all out of P1
3. Use ABT_pool_push_threads to push them to P2

In the above code, should I call ABT_thread_set_associated_pool before pushing the ULTs into P2?

I also see the function ABT_thread_migrate_to_pool. Can it be used to replace 3? I see in the documentation "Request a migration of a work unit to a specific pool.", but nothing about when this migration would happen. In the case of non-running ULTs, would this migration happen immediately?

[shintaro-iwasaki]

Thanks for your question, and sorry for the late reply.

Briefly speaking, I would do the following:

while (1) {
	ABT_thread threads[NUM_THREADS]; // Say NUM_THREADS = 64
	size_t num = 0;
	ABT_pool_pop_threads(pool1, threads, NUM_THREADS, &num);
	if (num == 0)
		break;
	ABT_pool_push_threads(pool2, threads, num);
}

Some details:

1. ABT_pool_get_size(): if you need it to check only the emptiness of the pool, ABT_pool_is_empty() is recommended.
2. Depending on the overhead of ABT_pool_pop_threads(), I would recommend using a statically-sized array (in the example, 64 elements). If you want to minimize the number of Argobots calls, please use ABT_pool_get_size() and dynamically allocate an array.
3. ABT_thread_set_associated_pool() is not necessary. It is to set a parent pool of a ULT and automatically updated when you push a ULT to another pool.
4. It's a migration "request" so the request can be ignored. I don't recommend this migration request feature because this is for backward compatibility (although I guess no one is using this feature in the world). Migration timing is not "specified", and Argobots kept a certain implementation-dependent timing (that I forgot) just in cases where someone keeps using it.

[mdorier]

Thanks for your answer!

I realized that if there are ULTs that are blocked (total_size of the pool > size of the pool), they won't be popped by ABT_pool_pop_threads. Assuming I can ensure that the pool is not used by any ES (so any blocked ULT are guaranteed not to be executing right now), would you see a way to also transfer these ULTs to pool2? (the difficulty is that I don't see any way to get a handle on them to begin with).

[shintaro-iwasaki]

Unfortunately, there's no way to pop ULTs that are blocked since internally, ABT_pool does not hold a list of blocked ULTs. I cannot come up with any good solution for this; creating a wrapper that can handle multiple ABT_pools instead of transferring ULTs in one pool to another might be simpler and easier.