infiniband soft_RoCE implement #1603

FlashBarryAllen · 2024-10-04T07:25:28Z

Soft-RoCE is the software implementation of the RDMA transport. It provides a high-performance, low-latency communication mechanism for applications that require direct memory access (DMA) between nodes.

Key features and benefits of Soft-RoCE:

Kernel Integration: Since Red Hat Enterprise Linux 7.4, the Soft-RoCE driver has been merged into the kernel, making it a standard component of the operating system.
User-Space Driver: The user-space driver is also integrated into the rdma-core package, providing a consistent API for applications to interact with the RDMA transport.
Performance: Soft-RoCE offers high performance and low latency, making it suitable for applications that require fast, reliable communication.
Flexibility: It can be used in a variety of network topologies, including Ethernet, Infiniband, and RoCE.
Soft-RoCE is also known as RXE. This is an alternative name for the technology.

And the wireshark has supported rdma, I think our pcapp should syn this feature.

codecov · 2024-10-04T08:16:15Z

Codecov Report

Attention: Patch coverage is 64.22764% with 88 lines in your changes missing coverage. Please review.

Project coverage is 83.03%. Comparing base (ae8652f) to head (dd63809).
Report is 2 commits behind head on dev.

Files with missing lines	Patch %	Lines
Packet++/src/InfiniBandLayer.cpp	52.05%	69 Missing and 1 partial ⚠️
Tests/Packet++Test/Tests/InfiniBandTests.cpp	78.08%	16 Missing ⚠️
Packet++/header/InfiniBandLayer.h	90.90%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##              dev    #1603      +/-   ##
==========================================
- Coverage   83.13%   83.03%   -0.10%     
==========================================
  Files         276      279       +3     
  Lines       47315    47561     +246     
  Branches     9520     9540      +20     
==========================================
+ Hits        39337    39494     +157     
- Misses       7091     7177      +86     
- Partials      887      890       +3

Flag	Coverage Δ
fedora40	`75.09% <48.29%> (-0.16%)`	⬇️
macos-13	`80.57% <58.54%> (-0.11%)`	⬇️
macos-14	`80.57% <58.54%> (-0.11%)`	⬇️
macos-15	`80.54% <58.54%> (-0.11%)`	⬇️
mingw32	`70.60% <42.85%> (-0.26%)`	⬇️
mingw64	`70.58% <42.85%> (-0.24%)`	⬇️
npcap	`84.99% <57.03%> (-0.21%)`	⬇️
rhel94	`74.94% <48.32%> (-0.13%)`	⬇️
ubuntu2004	`58.57% <40.12%> (-0.10%)`	⬇️
ubuntu2004-zstd	`58.71% <40.12%> (-0.10%)`	⬇️
ubuntu2204	`74.87% <47.97%> (-0.16%)`	⬇️
ubuntu2204-icpx	`61.40% <42.74%> (-0.08%)`	⬇️
ubuntu2404	`75.10% <48.29%> (-0.16%)`	⬇️
unittest	`83.03% <64.22%> (-0.10%)`	⬇️
windows-2019	`85.04% <57.03%> (-0.21%)`	⬇️
windows-2022	`85.06% <57.03%> (-0.21%)`	⬇️
winpcap	`85.02% <57.03%> (-0.20%)`	⬇️
xdp	`50.21% <47.97%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

tigercosmos

Thanks for the contribution. I take a quick look. Seems that the code is not formatted, and missing documents. Please take a look at https://pcapplusplus.github.io/community#contribute. Also, please try to follow the coding style with the other places in the repo.

Packet++/header/InfiniBandLayer.h

Packet++/header/ProtocolType.h

Packet++/src/InfiniBandLayer.cpp

Packet++/header/InfiniBandLayer.h

Tests/Packet++Test/Tests/InfiniBandTests.cpp

Packet++/CMakeLists.txt

Packet++/header/InfiniBandLayer.h

Packet++/header/ProtocolType.h

Tests/Packet++Test/main.cpp

seladb · 2024-10-17T08:05:14Z

@FlashBarryAllen in order to review this PR I'd like to know the protocol better. I did some reading and here is what I found. Can you please confirm these are good sources to understand the protocol?

What is RoCE? https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet - my understanding is that you're implementing RoCE v2, is that correct?
How is InfiniBand related to RoCE? https://www.hpcwire.com/2010/04/22/roce_an_ethernet-infiniband_love_story/
RoCE v2 InfiniBand packet format: https://enterprise-support.nvidia.com/s/article/rocev2-cnp-packet-format-example
A few pcap examples: https://www.zhaocs.info/soft-roce%E5%88%9D%E4%BD%93%E9%AA%8C.html Do you think you can provide more examples?

Packet++/CMakeLists.txt

Packet++/header/ProtocolType.h

seladb · 2024-10-17T08:08:24Z

Tests/Packet++Test/PacketExamples/InfinibandPacket.dat

Can you also provide a pcap/pcapng file so it's easier to open in Wireshark?

ok. Here's the pcap of rocev2.

Where is the pcap file? 🤔

sry, the file is too big to upload. I provide a small file of the roce.

You can extract only the packet in InfiniBandPacket.dat and put it in a pcap file, then add the pcap file to the repo

Packet++/src/UdpLayer.cpp

Tests/Packet++Test/CMakeLists.txt

Packet++/header/InfiniBandLayer.h

FlashBarryAllen · 2024-10-19T02:04:02Z

@FlashBarryAllen in order to review this PR I'd like to know the protocol better. I did some reading and here is what I found. Can you please confirm these are good sources to understand the protocol?

What is RoCE? https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet - my understanding is that you're implementing RoCE v2, is that correct?

How is InfiniBand related to RoCE? https://www.hpcwire.com/2010/04/22/roce_an_ethernet-infiniband_love_story/

RoCE v2 InfiniBand packet format: https://enterprise-support.nvidia.com/s/article/rocev2-cnp-packet-format-example

A few pcap examples: https://www.zhaocs.info/soft-roce%E5%88%9D%E4%BD%93%E9%AA%8C.html Do you think you can provide more examples?

@seladb

Yes, you are right. I just implementing RoCE v2. And we can found RDMA including RoCE v1 and RoCE v2. BTW, RDMA is consist of InfiniBand, RoCE v1, RoCE v2 and iWARP.

And I implement RoCE v2 by referring to linux.
https://github.com/torvalds/linux/tree/master/drivers/infiniband

seladb · 2024-10-19T07:31:47Z

@FlashBarryAllen in order to review this PR I'd like to know the protocol better. I did some reading and here is what I found. Can you please confirm these are good sources to understand the protocol?

What is RoCE? https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet - my understanding is that you're implementing RoCE v2, is that correct?

How is InfiniBand related to RoCE? https://www.hpcwire.com/2010/04/22/roce_an_ethernet-infiniband_love_story/

RoCE v2 InfiniBand packet format: https://enterprise-support.nvidia.com/s/article/rocev2-cnp-packet-format-example

A few pcap examples: https://www.zhaocs.info/soft-roce%E5%88%9D%E4%BD%93%E9%AA%8C.html Do you think you can provide more examples?

@seladb

Yes, you are right. I just implementing RoCE v2. And we can found RDMA including RoCE v1 and RoCE v2. BTW, RDMA is consist of InfiniBand, RoCE v1, RoCE v2 and iWARP.

And I implement RoCE v2 by referring to linux.
https://github.com/torvalds/linux/tree/master/drivers/infiniband

Thank you @FlashBarryAllen ! Please see the PR comments above

seladb · 2024-10-23T05:08:13Z

@FlashBarryAllen there are a few more open comments, can you please take of them before I continue to review the PR?

FlashBarryAllen · 2024-10-26T12:17:53Z

@FlashBarryAllen there are a few more open comments, can you please take of them before I continue to review the PR?
@seladb ok, I'll finish the comments above.

3rdParty/OUIDataset/PCPP_OUIDataset.json

seladb · 2024-10-29T07:27:11Z

Tests/Packet++Test/PacketExamples/InfinibandPacket.dat

Where is the pcap file? 🤔

Packet++/header/InfiniBandLayer.h

Packet++/src/InfiniBandLayer.cpp

Packet++/header/InfiniBandLayer.h

Packet++/src/InfiniBandLayer.cpp

FlashBarryAllen · 2024-11-03T14:00:55Z

here's the pcap of rocev2@seladb

Packet++/header/InfiniBandLayer.h

Packet++/src/InfiniBandLayer.cpp

seladb · 2024-11-06T08:09:10Z

Packet++/header/InfiniBandLayer.h

+		void setSolicitedEvent(bool se) const;
+
+		/**
+		 * @return migreq which used to communicate migration state


What is migreq? 🤔

seladb · 2024-11-06T08:09:25Z

Packet++/header/InfiniBandLayer.h

+		bool getMigrationState() const;
+
+		/**
+		 * Set migreq


ditto: What is migreq?

Packet++/header/InfiniBandLayer.h

seladb · 2024-11-06T08:22:47Z

Tests/Packet++Test/Tests/InfiniBandTests.cpp

+	for (auto* curLayer = parsedPacket.getFirstLayer(); curLayer != nullptr; curLayer = curLayer->getNextLayer())
+	{
+		switch (curLayer->getProtocol())
+		{
+		case pcpp::Ethernet:
+		{
+			// now let's get the Ethernet layer
+			auto* ethernetLayer = parsedPacket.getLayerOfType<pcpp::EthLayer>();
+			PTF_ASSERT_NOT_NULL(ethernetLayer);
+			break;
+		}
+		case pcpp::VLAN:
+		{
+			// now let's get the Vlan layer
+			auto* vlanLayer = parsedPacket.getLayerOfType<pcpp::VlanLayer>();
+			PTF_ASSERT_NOT_NULL(vlanLayer);
+			break;
+		}
+		case pcpp::IPv4:
+		{
+			// let's get the IPv4 layer
+			auto* ipLayer = parsedPacket.getLayerOfType<pcpp::IPv4Layer>();
+			PTF_ASSERT_NOT_NULL(ipLayer);
+			break;
+		}
+		case pcpp::UDP:
+		{
+			// let's get the UDP layer
+			auto* udpLayer = parsedPacket.getLayerOfType<pcpp::UdpLayer>();
+			PTF_ASSERT_NOT_NULL(udpLayer);
+			PTF_ASSERT_EQUAL(udpLayer->getSrcPort(), 57236);
+			PTF_ASSERT_EQUAL(udpLayer->getDstPort(), 4791);
+			break;
+		}


All this is not needed. Please look at other tests. You can only keep the InfiniBand part

seladb · 2024-11-06T08:23:24Z

Tests/Packet++Test/Tests/InfiniBandTests.cpp

+			PTF_ASSERT_EQUAL(ibLayer->getSoliciteEvent(), false);
+			PTF_ASSERT_EQUAL(ibLayer->getMigrationState(), false);


Instead of PTF_ASSERT_EQUAL you can use PTF_ASSERT_FALSE

seladb · 2024-11-06T08:23:33Z

Tests/Packet++Test/Tests/InfiniBandTests.cpp

+			PTF_ASSERT_EQUAL(ibLayer->getFecn(), false);
+			PTF_ASSERT_EQUAL(ibLayer->getBecn(), false);


seladb · 2024-11-06T08:29:03Z

Tests/Packet++Test/Tests/InfiniBandTests.cpp

+	pcpp::EthLayer ethLayer1(pcpp::MacAddress("02:7d:fa:01:17:40"), pcpp::MacAddress("02:7d:fa:00:10:01"),
+	                         PCPP_ETHERTYPE_IP);
+	pcpp::IPv4Layer ipLayer1(pcpp::IPv4Address("192.168.0.1"), pcpp::IPv4Address("192.168.0.2"));
+	pcpp::UdpLayer udpLayer1(30502, 4791);
+	pcpp::InfiniBandLayer infinibandLayer1(12, 0, 0, 0, 65535, 17, 1, 5557091);
+
+	pcpp::Packet infinibandPacket1(100);
+	PTF_ASSERT_TRUE(infinibandPacket1.addLayer(&ethLayer1));
+	PTF_ASSERT_TRUE(infinibandPacket1.addLayer(&ipLayer1));
+	PTF_ASSERT_TRUE(infinibandPacket1.addLayer(&udpLayer1));
+	PTF_ASSERT_TRUE(infinibandPacket1.addLayer(&infinibandLayer1));
+
+	PTF_ASSERT_EQUAL(infinibandPacket1.getLayerOfType<pcpp::UdpLayer>()->getDataLen(), 20);
+	PTF_ASSERT_NOT_NULL(infinibandPacket1.getLayerOfType<pcpp::InfiniBandLayer>());
+	PTF_ASSERT_EQUAL(udpLayer1.getDstPort(), 4791);
+	PTF_ASSERT_EQUAL(infinibandLayer1.getOpcode(), 12);
+	PTF_ASSERT_EQUAL(infinibandLayer1.getPartitionKey(), 65535);
+	PTF_ASSERT_EQUAL(infinibandLayer1.getQueuePairNumber(), 17);
+	PTF_ASSERT_EQUAL(infinibandLayer1.getAck(), 1);
+	PTF_ASSERT_EQUAL(infinibandLayer1.getPacketSequenceNumber(), 5557091);


You can write a test like this: https://github.com/seladb/PcapPlusPlus/blob/master/Tests/Packet%2B%2BTest/Tests/DhcpV6Tests.cpp#L123-L124

Basically what it does is:

Create the InfiniBand layer

Load the packet from the InfiniBand.dat file and get the InfiniBand layer

Assert that the raw buffer length of the layer in (1) is equal to the raw buffer length of the layer in (2)

Assert that the raw buffer of the layer in (1) is equal to the raw buffer of the layer in (2) using PTF_ASSERT_BUF_COMPARE

seladb · 2024-11-06T08:29:34Z

Tests/Packet++Test/Tests/InfiniBandTests.cpp

Please add InfiniBandEditTest to test all of the setter methods

FlashBarryAllen added 3 commits September 10, 2024 16:21

add infinibandlayer rxe for pcapp

64cbd61

add bth definiction

9f6016f

Merge branch 'dev' into master

bb61530

FlashBarryAllen requested a review from seladb as a code owner October 4, 2024 07:25

fix C-cast and add override to code

3a7dfaf

fix pre-commit errors

05333c8

tigercosmos requested changes Oct 4, 2024

View reviewed changes

FlashBarryAllen added 3 commits October 5, 2024 10:12

fix macro and coding style

008eab7

IB toString use more efficient style

86d3a58

IB coding style fix

5cbc3f9

seladb reviewed Oct 11, 2024

View reviewed changes

FlashBarryAllen added 7 commits October 13, 2024 15:15

add documentation of IB

b6f3c04

add IB creation test case

b7a2a74

Merge branch 'dev' into master

5d2402e

fix IB code format

9579f68

Merge branch 'dev' into master

33308c1

doxygen fix

342367e

Merge branch 'master' of github.com:FlashBarryAllen/PcapPlusPlus

d9aaee5

seladb reviewed Oct 17, 2024

View reviewed changes

FlashBarryAllen added 4 commits October 19, 2024 22:25

Merge branch 'dev' into master

254231f

IB code style fix

09a9e92

Merge branch 'master' of github.com:FlashBarryAllen/PcapPlusPlus

451aa5c

Merge branch 'dev' into master

1f0935e

Merge branch 'dev' into master

0623403

IB add pcap

8a05842

seladb reviewed Oct 29, 2024

View reviewed changes

FlashBarryAllen added 3 commits November 3, 2024 21:56

Merge branch 'dev' into master

4a02168

fix IB argument type

e2d8f04

Merge branch 'master' of github.com:FlashBarryAllen/PcapPlusPlus

dd63809

seladb reviewed Nov 6, 2024

View reviewed changes

egecetin linked an issue Nov 6, 2024 that may be closed by this pull request

can pcapp support soft-roce rdma packet header parse? #1569

Open

Merge branch 'dev' into master

6db4619

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

infiniband soft_RoCE implement #1603

infiniband soft_RoCE implement #1603

FlashBarryAllen commented Oct 4, 2024

codecov bot commented Oct 4, 2024 •

edited

Loading

tigercosmos left a comment •

edited

Loading

seladb commented Oct 17, 2024

seladb Oct 17, 2024

FlashBarryAllen Oct 26, 2024

seladb Oct 29, 2024

FlashBarryAllen Nov 3, 2024

seladb Nov 6, 2024

FlashBarryAllen commented Oct 19, 2024

seladb commented Oct 19, 2024

seladb commented Oct 23, 2024

FlashBarryAllen commented Oct 26, 2024

seladb Oct 29, 2024

FlashBarryAllen commented Nov 3, 2024

seladb Nov 6, 2024

seladb Nov 6, 2024

seladb Nov 6, 2024

seladb Nov 6, 2024

seladb Nov 6, 2024

seladb Nov 6, 2024

seladb Nov 6, 2024

		PTF_ASSERT_EQUAL(ibLayer->getSoliciteEvent(), false);
		PTF_ASSERT_EQUAL(ibLayer->getMigrationState(), false);

		PTF_ASSERT_EQUAL(ibLayer->getFecn(), false);
		PTF_ASSERT_EQUAL(ibLayer->getBecn(), false);

infiniband soft_RoCE implement #1603

Are you sure you want to change the base?

infiniband soft_RoCE implement #1603

Conversation

FlashBarryAllen commented Oct 4, 2024

codecov bot commented Oct 4, 2024 • edited Loading

Codecov Report

tigercosmos left a comment • edited Loading

Choose a reason for hiding this comment

seladb commented Oct 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FlashBarryAllen commented Oct 19, 2024

seladb commented Oct 19, 2024

seladb commented Oct 23, 2024

FlashBarryAllen commented Oct 26, 2024

Choose a reason for hiding this comment

FlashBarryAllen commented Nov 3, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Oct 4, 2024 •

edited

Loading

tigercosmos left a comment •

edited

Loading