use of H5S_SELECT_OR in H5Sselect_hyperslab()

[wkliao]

Affected cases:

• -a hdf5 -x canonical
• -a hdf5 -x log
• -a hdf5_md -x canonical

When using H5S_SELECT_OR in H5Sselect_hyperslab() to take a union of multiple
hyperslabs, HDF5 library will flatten all hyperslabs and sort the selected elements into
an increasing order. Therefore, it is not possible to create a filespace that follows the
same order of write requests using H5Sselect_hyperslab(). For example, if a process
makes 2 write requests, the 1st one is the 1st column of a MxN 2D array and the 2nd
request is the 2nd column, then H5S_SELECT_OR will create a hyperslab of Mx2.
In other words, one cannot write the 1st column followed by the 2nd column in a single
HDwrite call. See an example in tests/select_or.c.

In the current implementation, use of H5S_SELECT_OR appears in

• e3sm_io_driver_hdf5::varn_expand() and
• e3sm_io_driver_hdf5::post_varn()

The advantage of using the union of selected spaces is so we can create a single file
space per dataset and make a single H5Dwrite call to write the dataset, which also
allows us to use MPI collective mode. Otherwise, because the number of noncontiguous
write requests can be different among processes, we will have to call H5Dwrite one per
noncontiguous request without being able to use the collective I/O.

One solution will be to flatten all write requests and merge them into an increasing order
in the file space. This will also require to flatten the memory space of the user buffers
and move the memory buffers corresponding to their file spaces during the sorting.

[wkliao]

Thanks to @dqwu's confirmation, PIO's dataset/box re-arranger does this correctly by sorting the start-count pairs together with the user buffers into an increasing order. Thus Scorpio can reply on this fact and safely take a union of multiple hyperslabs.