mercury-hpc · soumagne · Dec 20, 2023 · Sep 21, 2023 · Dec 1, 2023
diff --git a/Documentation/CHANGES.md b/Documentation/CHANGES.md
@@ -1,67 +1,30 @@
 ## Summary
 
-This version brings bug fixes and updates to our v2.3.0 release.
+This is a preview release of the v2.4.0 release.
 
 ## New features
 
-- __[HG info]__
-    - Add support for CSV and JSON output formats
+- __[NA]__
+    - Add init info version compatibility wrappers
+    - Bump NA version to v4.1.0
+    - Add support for `traffic_class` init info (only supported by ofi plugin)
 - __[HG/NA Perf Test]__
-    - Enable sizes to be passed using k/m/g qualifiers
+    - Add `-f`/`--hostfile` option to select hostfile to write to / read from
+    - Add `-T`/`--tclass` option to select trafic class
+    - Autodetect MPI implementation in perf utilities
+        - MPI can now be autodetected and dynamically loaded in utilities, even if `MERCURY_TESTING_ENABLE_PARALLEL` was turned off. If `MERCURY_TESTING_ENABLE_PARALLEL` is turned on, tests remain manually linked against MPI as they used to be.
 - __[NA OFI]__
-    - Add `tcp_rxm` alias for `tcp;ofi_rxm`
-    - Find CXI `svc_id` or `vni` if `auth_key` components have zeros (e.g., `auth_key=0:0`)
-        - Add VNI index for `SLINGSHOT_VNIS` discovery as extra auth_key parameter
+    - Attempt to distribute multi-NIC domains based on selected CPU ID
+    - Support selection of traffic classes (single class per NA class)
 
 ## Bug fixes
 
-- __[HG/NA]__
-    - Fix potential race when checking secondary completion queue
-- __[HG]__
-    - Prevent multiple threads from entering `HG_Core_progress()`
-        - Add `HG_ALLOW_MULTI_PROGRESS` CMake option to control behavior (`ON` by default)
-        - Disable `NA_HAS_MULTI_PROGRESS` if `HG_ALLOW_MULTI_PROGRESS` is `ON`
-    - Fix expected operation count for handle to be atomic
-        - Expected operation count can change if extra RPC payload must be transferred
-    - Let poll events remain private to HG poll wait
-        - Prevent a race when multiple threads call progress and `HG_ALLOW_MULTI_PROGRESS` is `OFF`
-    - Separate internal list from user created list of handles
-        - Address an issue where `HG_Context_unpost()` would unnecessarily wait
-- __[HG Core]__
-    - Cache disabled response info in proc info
-    - Add `HG_Core_registered_disable(d)_response()` routines
-    - Refactor and optimize self RPC code path
-    - Add additional logging of refcount/expected op count
-    - Fixes for self RPCs with no response
 - __[HG Util]__
-    - Prevent locking in `hg_request_wait()`
-        - Concurrent progress in multi-threaded scenarios on the same context could complete another thread's request and let a thread blocked in progress
-- __[HG Perf]__
-    - Fix tests to be run in parallel with any communicator size
-- __[HG Test]__
-    - Ensure affinity of class thread is set
-    - Add concurrent multi RPC test
-    - Add multi-progress test
-    - Add multi-progress test with handle creation
-    - Refactoring of unit test cleanup
-- __[NA]__
-    - Fix memory leak on `NA_Get_protocol_info()`
+    - Add missing prototype to `hg_atomic_fence()` definition
 - __[NA OFI]__
-    - Fix `na_ofi_get_protocol_info()` not returning `opx` protocol
-        - Refactor `na_ofi_getinfo()` to account for `NA_OFI_PROV_NULL` type
-        - Ensure there are no duplicated entries
-    - Refactor parsing of init info strings and fix OPX parsing
-    - Simplify parsing of some address strings
-    - Bump default CQ size to have a maximum depth of 128k entries
-    - Remove sockets as the only provider on macOS
-    - Remove send afer send tagged msg ordering
-    - Ensure that `rx_ctx_bits` are not set if SEP is not used
-    - Set CXI domain ops w/ slingshot 2.2 to prevent from potential memory corruptions
+    - Remove excessive MR count warning message
 - __[NA Perf]__
-    - Prevent tests from being run as parallel tests
-- __[CMake]__
-    - Pass `INSTALL_NAME_DIR` through target properties
-        - This fixes an issue seen on macOS where libraries would not be found using `@rpath`
+    - Ensure perf tests wait on send completion
 
 ## :warning: Known Issues
 

diff --git a/Documentation/CHANGES_v2.3.1.md b/Documentation/CHANGES_v2.3.1.md
@@ -0,0 +1,69 @@
+## Summary
+
+This version brings bug fixes and updates to our v2.3.0 release.
+
+## New features
+
+- __[HG info]__
+    - Add support for CSV and JSON output formats
+- __[HG/NA Perf Test]__
+    - Enable sizes to be passed using k/m/g qualifiers
+- __[NA OFI]__
+    - Add `tcp_rxm` alias for `tcp;ofi_rxm`
+    - Find CXI `svc_id` or `vni` if `auth_key` components have zeros (e.g., `auth_key=0:0`)
+        - Add VNI index for `SLINGSHOT_VNIS` discovery as extra auth_key parameter
+
+## Bug fixes
+
+- __[HG/NA]__
+    - Fix potential race when checking secondary completion queue
+- __[HG]__
+    - Prevent multiple threads from entering `HG_Core_progress()`
+        - Add `HG_ALLOW_MULTI_PROGRESS` CMake option to control behavior (`ON` by default)
+        - Disable `NA_HAS_MULTI_PROGRESS` if `HG_ALLOW_MULTI_PROGRESS` is `ON`
+    - Fix expected operation count for handle to be atomic
+        - Expected operation count can change if extra RPC payload must be transferred
+    - Let poll events remain private to HG poll wait
+        - Prevent a race when multiple threads call progress and `HG_ALLOW_MULTI_PROGRESS` is `OFF`
+    - Separate internal list from user created list of handles
+        - Address an issue where `HG_Context_unpost()` would unnecessarily wait
+- __[HG Core]__
+    - Cache disabled response info in proc info
+    - Add `HG_Core_registered_disable(d)_response()` routines
+    - Refactor and optimize self RPC code path
+    - Add additional logging of refcount/expected op count
+    - Fixes for self RPCs with no response
+- __[HG Util]__
+    - Prevent locking in `hg_request_wait()`
+        - Concurrent progress in multi-threaded scenarios on the same context could complete another thread's request and let a thread blocked in progress
+- __[HG Perf]__
+    - Fix tests to be run in parallel with any communicator size
+- __[HG Test]__
+    - Ensure affinity of class thread is set
+    - Add concurrent multi RPC test
+    - Add multi-progress test
+    - Add multi-progress test with handle creation
+    - Refactoring of unit test cleanup
+- __[NA]__
+    - Fix memory leak on `NA_Get_protocol_info()`
+- __[NA OFI]__
+    - Fix `na_ofi_get_protocol_info()` not returning `opx` protocol
+        - Refactor `na_ofi_getinfo()` to account for `NA_OFI_PROV_NULL` type
+        - Ensure there are no duplicated entries
+    - Refactor parsing of init info strings and fix OPX parsing
+    - Simplify parsing of some address strings
+    - Bump default CQ size to have a maximum depth of 128k entries
+    - Remove sockets as the only provider on macOS
+    - Remove send afer send tagged msg ordering
+    - Ensure that `rx_ctx_bits` are not set if SEP is not used
+    - Set CXI domain ops w/ slingshot 2.2 to prevent from potential memory corruptions
+- __[NA Perf]__
+    - Prevent tests from being run as parallel tests
+- __[CMake]__
+    - Pass `INSTALL_NAME_DIR` through target properties
+        - This fixes an issue seen on macOS where libraries would not be found using `@rpath`
+
+## :warning: Known Issues
+
+- __[NA OFI]__
+    - [tcp/verbs;ofi_rxm] Using more than 256 peers requires `FI_UNIVERSE_SIZE` to be set.
diff --git a/Testing/common/na_test.c b/Testing/common/na_test.c
@@ -53,6 +53,9 @@ na_test_parse_options(
 static size_t
 na_test_parse_size(const char *str);
 
+static enum na_traffic_class
+na_test_tclass(const char *str);
+
 static char *
 na_test_gen_config(struct na_test_info *na_test_info, unsigned int i);
 
@@ -95,6 +98,7 @@ na_test_usage(const char *execname)
            "                         Default: any\n");
     printf("    -S, --self_send      Send to self\n");
     printf("    -k, --key            Pass auth key\n");
+    printf("    -T, --tclass         Traffic class to use\n");
     printf("    -l, --loop           Number of loops (default: 1)\n");
     printf("    -b, --busy           Busy wait\n");
     printf("    -y  --buf_size_min   Min buffer size (in bytes)\n");
@@ -203,6 +207,9 @@ na_test_parse_options(int argc, char *argv[], struct na_test_info *na_test_info)
             case 'f': /* hostfile */
                 na_test_info->hostfile = strdup(na_test_opt_arg_g);
                 break;
+            case 'T': /* tclass */
+                na_test_info->tclass = strdup(na_test_opt_arg_g);
+                break;
             default:
                 break;
         }
@@ -245,6 +252,30 @@ na_test_parse_size(const char *str)
         return 0;
 }
 
+/*---------------------------------------------------------------------------*/
+static enum na_traffic_class
+na_test_tclass(const char *str)
+{
+    if (strcmp(str, "best_effort") == 0)
+        return NA_TC_BEST_EFFORT;
+    else if (strcmp(str, "low_latency") == 0)
+        return NA_TC_LOW_LATENCY;
+    else if (strcmp(str, "bulk_data") == 0)
+        return NA_TC_BULK_DATA;
+    else if (strcmp(str, "dedicated_access") == 0)
+        return NA_TC_DEDICATED_ACCESS;
+    else if (strcmp(str, "scavenger") == 0)
+        return NA_TC_SCAVENGER;
+    else if (strcmp(str, "network_ctrl") == 0)
+        return NA_TC_NETWORK_CTRL;
+    else {
+        fprintf(stderr,
+            "Traffic class does not match: best_effort, low_latency, "
+            "bulk_data, dedicated_access, scavenger, network_ctrl\n");
+        return NA_TC_UNSPEC;
+    }
+}
+
 /*---------------------------------------------------------------------------*/
 static char *
 na_test_gen_config(struct na_test_info *na_test_info, unsigned int i)
@@ -480,6 +511,13 @@ NA_Test_init(int argc, char *argv[], struct na_test_info *na_test_info)
     na_init_info.max_expected_size = (size_t) na_test_info->max_msg_size;
     na_init_info.thread_mode =
         na_test_info->use_threads ? 0 : NA_THREAD_MODE_SINGLE;
+    if (na_test_info->tclass != NULL) {
+        na_init_info.traffic_class = na_test_tclass(na_test_info->tclass);
+        NA_TEST_CHECK_ERROR(na_init_info.traffic_class == NA_TC_UNSPEC, error,
+            ret, NA_PROTONOSUPPORT, "Unsupported traffic class");
+        if (na_test_info->mpi_info.rank == 0)
+            printf("# Using traffic class: %s\n", na_test_info->tclass);
+    }
 
     na_test_info->na_classes = (na_class_t **) malloc(
         sizeof(na_class_t *) * na_test_info->max_classes);
@@ -649,6 +687,10 @@ NA_Test_finalize(struct na_test_info *na_test_info)
         free(na_test_info->hostfile);
         na_test_info->hostfile = NULL;
     }
+    if (na_test_info->tclass != NULL) {
+        free(na_test_info->tclass);
+        na_test_info->tclass = NULL;
+    }
 
     na_test_mpi_finalize(&na_test_info->mpi_info);
 

diff --git a/Testing/common/na_test.h b/Testing/common/na_test.h
@@ -29,6 +29,7 @@ struct na_test_info {
     bool mpi_static;         /* MPI static comm */
     bool self_send;          /* Self send */
     char *key;               /* Auth key */
+    char *tclass;            /* Traffic class */
     int loop;                /* Number of loops */
     bool busy_wait;          /* Busy wait */
     size_t max_classes;      /* Max classes */

diff --git a/Testing/common/na_test_getopt.c b/Testing/common/na_test_getopt.c
@@ -14,7 +14,8 @@
 
 int na_test_opt_ind_g = 1;            /* token pointer */
 const char *na_test_opt_arg_g = NULL; /* flag argument (or value) */
-const char *na_test_short_opt_g = "hc:d:p:H:P:sSk:l:bC:X:VZ:y:z:w:x:mt:BRvMUf:";
+const char *na_test_short_opt_g =
+    "hc:d:p:H:P:sSk:l:bC:X:VZ:y:z:w:x:mt:BRvMUf:T:";
 /* clang-format off */
 const struct na_test_opt na_test_opt_g[] = {
     {"help", no_arg, 'h'},
@@ -44,6 +45,7 @@ const struct na_test_opt na_test_opt_g[] = {
     {"millionbps", no_arg, 'M'},
     {"no-multi-recv", no_arg, 'U'},
     {"hostfile", require_arg, 'f'},
+    {"tclass", require_arg, 'T'},
     {NULL, 0, '\0'} /* Must add this at the end */
 };
 /* clang-format on */

diff --git a/src/mercury_core.c b/src/mercury_core.c
@@ -1198,8 +1198,11 @@ hg_core_init(const char *na_info_string, hg_bool_t na_listen,
             HG_MAJOR(version), HG_MINOR(version));
 
         /* Get init info and overwrite defaults */
-        if (HG_VERSION_GE(version, HG_VERSION(2, 3)))
+        if (HG_VERSION_GE(version, HG_VERSION(2, 4)))
             hg_init_info = *hg_init_info_p;
+        else if (HG_VERSION_GE(version, HG_VERSION(2, 3)))
+            hg_init_info_dup_2_3(&hg_init_info,
+                (const struct hg_init_info_2_3 *) hg_init_info_p);
         else
             hg_init_info_dup_2_2(&hg_init_info,
                 (const struct hg_init_info_2_2 *) hg_init_info_p);

diff --git a/src/mercury_private.h b/src/mercury_private.h
@@ -17,8 +17,23 @@
 /*************************************/
 
 /* Previous versions of init info to keep compatiblity with older versions */
+struct hg_init_info_2_3 {
+    struct na_init_info_4_0 na_init_info;
+    na_class_t *na_class;
+    hg_uint32_t request_post_init;
+    hg_uint32_t request_post_incr;
+    hg_bool_t auto_sm;
+    const char *sm_info_string;
+    hg_checksum_level_t checksum_level;
+    hg_bool_t no_bulk_eager;
+    hg_bool_t no_loopback;
+    hg_bool_t stats;
+    hg_bool_t no_multi_recv;
+    hg_bool_t release_input_early;
+};
+
 struct hg_init_info_2_2 {
-    struct na_init_info na_init_info;
+    struct na_init_info_4_0 na_init_info;
     na_class_t *na_class;
     hg_uint32_t request_post_init;
     hg_uint32_t request_post_incr;
@@ -87,8 +102,11 @@ extern "C" {
  * Duplicate init info for ABI compatibility.
  */
 static HG_INLINE void
-hg_init_info_dup_2_2(struct hg_init_info *hg_init_info,
-    const struct hg_init_info_2_2 *hg_init_info_2_2);
+hg_init_info_dup_2_3(
+    struct hg_init_info *new_info, const struct hg_init_info_2_3 *old_info);
+static HG_INLINE void
+hg_init_info_dup_2_2(
+    struct hg_init_info *new_info, const struct hg_init_info_2_2 *old_info);
 
 /**
  * Increment bulk handle counter.
@@ -136,22 +154,40 @@ hg_bulk_op_pool_destroy(struct hg_bulk_op_pool *hg_bulk_op_pool);
 
 /*---------------------------------------------------------------------------*/
 static HG_INLINE void
-hg_init_info_dup_2_2(struct hg_init_info *hg_init_info,
-    const struct hg_init_info_2_2 *hg_init_info_2_2)
+hg_init_info_dup_2_3(
+    struct hg_init_info *new_info, const struct hg_init_info_2_3 *old_info)
+{
+    *new_info = (struct hg_init_info){.na_class = old_info->na_class,
+        .request_post_init = old_info->request_post_init,
+        .request_post_incr = old_info->request_post_incr,
+        .auto_sm = old_info->auto_sm,
+        .sm_info_string = old_info->sm_info_string,
+        .checksum_level = old_info->checksum_level,
+        .no_bulk_eager = old_info->no_bulk_eager,
+        .no_loopback = old_info->no_loopback,
+        .stats = old_info->stats,
+        .no_multi_recv = old_info->no_multi_recv,
+        .release_input_early = old_info->release_input_early};
+    na_init_info_dup_4_0(&new_info->na_init_info, &old_info->na_init_info);
+}
+
+/*---------------------------------------------------------------------------*/
+static HG_INLINE void
+hg_init_info_dup_2_2(
+    struct hg_init_info *new_info, const struct hg_init_info_2_2 *old_info)
 {
-    *hg_init_info =
-        (struct hg_init_info){.na_init_info = hg_init_info_2_2->na_init_info,
-            .na_class = hg_init_info_2_2->na_class,
-            .request_post_init = hg_init_info_2_2->request_post_init,
-            .request_post_incr = hg_init_info_2_2->request_post_incr,
-            .auto_sm = hg_init_info_2_2->auto_sm,
-            .sm_info_string = hg_init_info_2_2->sm_info_string,
-            .checksum_level = hg_init_info_2_2->checksum_level,
-            .no_bulk_eager = hg_init_info_2_2->no_bulk_eager,
-            .no_loopback = hg_init_info_2_2->no_loopback,
-            .stats = hg_init_info_2_2->stats,
-            .no_multi_recv = HG_FALSE,
-            .release_input_early = HG_FALSE};
+    *new_info = (struct hg_init_info){.na_class = old_info->na_class,
+        .request_post_init = old_info->request_post_init,
+        .request_post_incr = old_info->request_post_incr,
+        .auto_sm = old_info->auto_sm,
+        .sm_info_string = old_info->sm_info_string,
+        .checksum_level = old_info->checksum_level,
+        .no_bulk_eager = old_info->no_bulk_eager,
+        .no_loopback = old_info->no_loopback,
+        .stats = old_info->stats,
+        .no_multi_recv = HG_FALSE,
+        .release_input_early = HG_FALSE};
+    na_init_info_dup_4_0(&new_info->na_init_info, &old_info->na_init_info);
 }
 
 #ifdef __cplusplus

diff --git a/src/na/na.c b/src/na/na.c
@@ -294,8 +294,9 @@ na_info_parse(
     na_info = (struct na_info *) malloc(sizeof(struct na_info));
     NA_CHECK_SUBSYS_ERROR(cls, na_info == NULL, error, ret, NA_NOMEM,
         "Could not allocate NA info struct");
-    *na_info = (struct na_info){
-        .host_name = NULL, .protocol_name = NULL, .na_init_info = NULL};
+    *na_info = (struct na_info){.host_name = NULL,
+        .protocol_name = NULL,
+        .na_init_info = NA_INIT_INFO_INITIALIZER};
 
     /* Copy info string and work from that */
     input_string = strdup(info_string);
@@ -764,7 +765,13 @@ NA_Initialize_opt2(const char *info_string, bool listen, unsigned int version,
         NA_LOG_SUBSYS_DEBUG(cls, "Init info version used: v%d.%d",
             NA_MAJOR(version), NA_MINOR(version));
 
-        na_info->na_init_info = na_init_info;
+        /* Get init info and overwrite defaults */
+        if (NA_VERSION_GE(version, NA_VERSION(4, 1)))
+            na_info->na_init_info = *na_init_info;
+        else
+            na_init_info_dup_4_0(&na_info->na_init_info,
+                (const struct na_init_info_4_0 *) na_init_info);
+
         na_private_class->na_class.progress_mode = na_init_info->progress_mode;
     }