RDMA implementation

[mond77]

Development Progress

https://github.com/mond77/ibv

Communication paradigm

The existing usages of RDMA lead to a dilemma between performance and redesign cost. They either directly replace socket-based send/receive primitives with the corresponding RDMA counterpart (server-reply), which only achieves moderate performance improvement; or push performance further by using one-sided RDMA operations to totally bypass the server (server-bypass), at the cost of redesigning the software.

Instead of server-bypass or server-reply paradigm, we could design a new RDMA paradigm that fits for RPC of sealfs.

Related reference papers:

1. https://ieeexplore.ieee.org/document/8588372 ('client read' might be compared with 'server write' in the future)

Learning RDMA

1. The concepts of RDMA: https://www.zhihu.com/column/c_1231181516811390976
2. The ibverbs API: https://linux.die.net/man/3/ibv_create_qp
3. RDMA blogs: http://www.rdmamojo.com/

[mond77]

I have tried using async-rdma crate to implement rpc of sealfs, but the performance of async-rdma in Soft-RoCE was lower than rpc based in TCP. And it comes to encounter some implicit problem in much currency rpc. If you are interested in it, you can see the branch.
While trying to implement rpc in a way of RDMA send/recv, the problem unsolved was listed below:

• the data length limit for the fixed memory;
• the Insufficient RQE when Large number of send requests

[mond77]

use a large block of memory as receive buffer.
use read/write instead of send/recv.

[mond77]

The remote end can only passively receive data, so a large block of memory is used as the receive buffer.

The local end:
The first option:
Send the data of each request in a WR (work request) with a S/G (scatter/gather) list.
The advantage is that there is no limit to the number of requests sent, but memory pooling needs to be implemented.

The second option:
Allocate a large section of memory as a send buffer like TCP.
The advantages include ease of memory management, but possible queue head blocking problems, and requests for large blocks of data can significantly reduce IOPS.