diff --git a/source/elements/oneCCL/source/spec/collective_operations.rst b/source/elements/oneCCL/source/spec/collective_operations.rst index 3987ded88d..97e9479344 100644 --- a/source/elements/oneCCL/source/spec/collective_operations.rst +++ b/source/elements/oneCCL/source/spec/collective_operations.rst @@ -15,6 +15,7 @@ oneCCL specification defines the following collective communication operations: - `Broadcast`_ - `Reduce`_ - `ReduceScatter`_ +- `PointToPoint`_ These operations are collective, meaning that all participants (ranks) of oneCCL communicator should make a call. The order of collective operation calls should be the same across all ranks. @@ -398,6 +399,142 @@ return ``event`` an object to track the progress of the operation +.. _PointToPoint: + +Point-To-Point Operations +************************* + +OneCCL specification defines the following point-to-point operations: + +* Send +* Recv + +In point-to-point communication, two ranks participate in the communication so when a process sends data to a peer rank, +the peer rank needs to post a ``recv`` call with the same datatype and count as the sending rank. + +The current specification only supports blocking ``send`` and ``recv`` and does not support for multiple ``send`` +and ``receive`` operations to proceed concurrently. + +In the ``send`` operation, the peer specifies the destination process, while in the recv operation peer specifies the source process. + +As with the collective operations, the communicator can perform communication operations on host or device memory buffers +depending on the device used to create the communicator. Additionally, communication operations accept an execution +context (stream) and may accept a vector of events on which the communication operation should depend, that is, input dependencies. +The output event object provides the ability to track the operation's progress. + +.. note:: Support for the handling of input events is optional. + +BufferType is used below to define the C++ type of elements in communication operations' data buffers +(``buf``, ``send_buf``, and ``recv_buf``). At least the following types should be supported: ``[u]int{8/16/32/64}_t, float, double``. +The explicit datatype parameter enable data types that cannot be inferred from the function arguments. +For more information, see Custom Datatypes. + +The communication operation accepts a stream object. If a communicator is created from ``native_device_type``, +then the stream translates to ``native_stream_type`` created from the corresponding device. + +The communication operation may accept attribute objects. If that parameter is missed, then the default attribute object is used +(``default__attr``). The default attribute object is provided by the library. +For more information, see Operation Attributes. + +If the arguments provided to a communication operation call do not comply with the requirements of the operation, +the behavior is undefined, unless otherwise specified. + +Send +^^^^ + +A blocking point-to-point communication operation that sends the data in a buf to a peer rank. + +.. code:: cpp + + template & deps = {}); + + event CCL_API send(void *buf, + size_t count, + datatype dtype, + int peer, + const communicator &comm, + const stream &stream, + const pt2pt_attr &attr = default_pt2pt_attr, + const vector_class &deps = {}); + +buf + the buffer with count elements of ``dtype`` that contains the data to be sent +count + the number of elements of type dtype in buf +dtype + the datatype of elements in buf + must be skipped if ``BufferType`` can be inferred + otherwise must be passed explicitly +peer + the destination rank +comm + the communicator that defines a group of ranks for the operation +stream + the stream associated with the operation +attr + optional attributes to customize the operation +deps + an optional vector of the events that the operation should depend on + +return event + an object to track the progress of the operation + +Recv +^^^^^ + +A blocking point-to-point communication operation that receives the data in a buf from a peer rank. + +.. code:: cpp + + template &deps = {}); + + event CCL_API send(void *buf, + size_t count, + datatype dtype, + int peer, + const communicator &comm, + const stream &stream, + const pt2pt_attr &attr = default_pt2pt_attr, + const vector_class &deps = {}); + +buf [out] + the buffer with count elements of dtype that contains the data to be sent +count + the number of elements of type dtype in buf +dtype + the datatype of elements in buf + must be skipped if ``BufferType`` can be inferred + otherwise must be passed explicitly +peer + the destination rank +comm + the communicator that defines a group of ranks for the operation +stream + The stream associated with the operation +attr + optional attributes to customize the operation +deps + an optional vector of the events that the operation should depend on + +return event + object to track the progress of the operation + + + .. note:: See also: