coll: add coll_group to collective interfaces #7103

Store num_local and num_external in MPIR_Comm. Along with internode_table, they help construct internode subgroups.

This is the same as num_external.

As the title.

It does not take many instructions to calculate pof2 on the fly. Use of hard coded pof2 prevents collective algorithms to be used for non-trivial coll_group.

Lightweight struct to describe sub-groups of a communicator. They intend to replace the subcomms. Preset a set of reserved subgroups to simplify common usages such as intranode group and crossnode group. Since we only expect limited number of dynamic subgroups and they should always be push/pop'ed within the scope, we don't need many dynamic slots.

Group collectives will have non-trivial coll_group that alter the rank and size of the communicator. Thease macros and functions will facilitate it.

Add coll_group, index to comm->subgroups[], to all collectives except neighborhood collectives.

Assuming the device layer collectives are not able to handle non-trivial coll_group, always fallback when coll_group != MPIR_SUBGROUP_NONE, for now. Also normalize the code style to use the fallback label. We should always fallback to mpir impl routines rather than the netmod routines (composition_beta). The composition_beta may fallback in the future when netmod coll become fancy, resulting in deadloop.

Make csel coll_group aware.

Use coll_group=MPIR_SUBGROUP_THREADCOMM for threadcomm collectives. This allows compositional collectives under threadcomm.

We call MPIR_Comm_is_parent_comm to prevent recursively entering compositional algorithms such as the _smp algorithms. Check coll_group as well as we will switch to use subgroup rather than subcomms. Also check num_external directly for trivial comm. Subcomms and comm->hierarchy_kind will be removed in the future.

Use MPIR_COLL_RANK_SIZE if the algorithm is topology neutral. Use MPIR_COLL_RANK_SIZE_NO_GROUP if the algorithm is topology dependent. It adds an assertion on coll_group == MPIR_SUBGROUPS_NONE since coll_group may alter the topology assumptions. Intercomm does not work with non-zero coll_group.

Replace the usage of subcomms with subgroups.

When root is not local rank 0, instead of adding a extra intra-node send/recv or bcast, construct an inter group that includes the root process.

Directly use information from MPIR_Process rather than from nodecomm in MPIR_Process. One step toward removing subcomms.

Use a single "cached_tree" rather than 3 different fields for each tree type.

The topology-aware tree utilities need check coll_group for correct world ranks.

Some algorithm, e.g. Allgather recexch, caches comm size-related info in communicator, thus won't work with none trivial coll_group. Add a restriction so it will fallback when coll_group != MPIR_SUBGROUP_NONE.

All subgroup collectives should use the same tag within the parent collectives. This is because all processes in the communicator has to agree on the tag to use, but group collectives may not involve all processes. It is okay to use the same tag as long as the group collectives are always issued in order. This is the case since all group collectives are spawned under a parent collective, which has to obey the non-overlapping rule.

Because the compiler can't figure out the arithmetic, it is warning: ‘MPIC_Waitall’ accessing 8 bytes in a region of size 0 [-Wstringop-overflow=] Refactor to suppress warning and for better readability.

Commit ba1b4dd left an empty branch that should be removed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

coll: add coll_group to collective interfaces #7103

coll: add coll_group to collective interfaces #7103

Commits on Sep 14, 2024

coll: add coll_group to collective interfaces #7103

Are you sure you want to change the base?

coll: add coll_group to collective interfaces #7103

Commits on Sep 14, 2024