Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Linux XDP support #3849

Merged
merged 179 commits into from
Apr 25, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
179 commits
Select commit Hold shift + click to select a range
420590b
built on Linux
ami-GS May 14, 2023
d63fd7f
ne files
ami-GS May 14, 2023
8cb383d
renaming
ami-GS May 14, 2023
a32d79b
split socket funcs
ami-GS May 14, 2023
ba2b925
refactor from win
ami-GS May 14, 2023
7667acc
build from linux
ami-GS May 14, 2023
5cfd379
more on linux
ami-GS May 14, 2023
05d191a
rename and generate
ami-GS May 14, 2023
11773d7
use libbpf
ami-GS May 19, 2023
7e71548
oops
ami-GS May 19, 2023
2d79d2e
rebase and add new member
ami-GS May 20, 2023
2ca3399
unify CxPlatTryAddSocket
ami-GS May 21, 2023
74ab9cb
pass datapath tests
ami-GS Jun 4, 2023
cf46e48
integrate Ubuntu 22.04 default libbpf v0.5.0
ami-GS Jun 6, 2023
ef8bd89
temporally
ami-GS Jun 27, 2023
8647673
multithread. all test passed
ami-GS Jul 5, 2023
1f2b3fd
secnetperf works
Jul 22, 2023
3483760
update build scripts
ami-GS Sep 11, 2023
b4656a4
nit update
ami-GS Sep 11, 2023
b3805f0
add duonic installer script for linux
ami-GS Sep 11, 2023
3ae0c0a
add xdp-tools directory
ami-GS Sep 11, 2023
1b74349
Merge commit '707c1fab' into dev/daiki/xdp_split_functional
ami-GS Sep 11, 2023
008ab54
Merge commit '88f0269' into dev/daiki/xdp_split_functional
ami-GS Sep 11, 2023
20961da
Merge commit 'dc64ccb47adf847d01f171bd3ccab1818b5d92d8' into dev/daik…
ami-GS Sep 12, 2023
17d9fa0
Merge commit 'f596ae5313a3b925556890482a2839a81c9d0392' into dev/daik…
ami-GS Sep 12, 2023
9a6bcae
Merge commit 'b6633eace10ed493cb234f86e18cc6d9d7526417' into dev/daik…
ami-GS Sep 12, 2023
a2c7477
Merge commit '8af9c3b214e83ea3146f88c072a0220a22a48ee6' into dev/daik…
ami-GS Sep 12, 2023
846aa31
Merge commit 'c885d5f1fdab56a8630aa1ec8a0b0bc41da291ca' into dev/daik…
ami-GS Sep 12, 2023
59aa678
Merge commit 'e6d504407e3a0e739d3da4c3393da43af61ae69f' into dev/daik…
ami-GS Sep 12, 2023
170279d
cleanup
ami-GS Sep 12, 2023
aa8f0d9
fix Windows build
ami-GS Sep 12, 2023
457dd4a
fix windows xdp build
ami-GS Sep 12, 2023
df4c7bd
Merge branch 'main' into dev/daiki/xdp_split_functional
ami-GS Sep 13, 2023
1ffded4
add Github action
ami-GS Sep 13, 2023
d57341d
nit fix
ami-GS Sep 13, 2023
421ea69
fix dependencies for Install-DuoNic
ami-GS Sep 13, 2023
c57ee78
add udpate
ami-GS Sep 13, 2023
ee0e042
add linux xdp dependencies to docker file
ami-GS Sep 15, 2023
a744683
fix installation script
ami-GS Sep 15, 2023
7dc44cd
initialize submodules/xdp-tools recursively
ami-GS Sep 21, 2023
b5a846b
Merge commit 'f9d26ac' into dev/daiki/xdp_split_functional
ami-GS Mar 6, 2024
027e886
Merge commit 'ae3db5cd991ea872efcf9e32dbe63f4f203245af' into dev/daik…
ami-GS Mar 14, 2024
2f1687f
Merge commit '284738a861272c63f31d900c490e9d5e856c8ac2' into dev/daik…
ami-GS Mar 14, 2024
3d1e986
Merge commit '4ef3731d82757451f9b62df0e7b41f32c87a1392' into dev/daik…
ami-GS Mar 14, 2024
6cf52f4
Merge commit '649f971b1bcce59ce0aba2d30a199d19dd91a4d2' into dev/daik…
ami-GS Mar 14, 2024
2f21a23
Merge commit '8e1eb1a74b6baed0af7dad899b31baf970051d26' into dev/daik…
ami-GS Mar 14, 2024
f569699
Merge commit '6e1c3d6af6ecade0979ec3430221c8920e562e91' into dev/daik…
ami-GS Mar 14, 2024
a122ae7
Merge commit 'fe9c2971a3abb432f92fff86664f1f131fe728b1' into dev/daik…
ami-GS Mar 14, 2024
bbe0af7
Merge commit 'e73ce1082364c697f89bfaf81c67b3158b9987c0' into dev/daik…
ami-GS Mar 14, 2024
e803c64
Merge commit '9483184dd39559f20daf57f95fdbcb14c367ff01' into dev/daik…
ami-GS Mar 14, 2024
6dd87ff
Merge commit 'cacfccae195ab36f91fd7e2c27ab56a863935ba3' into dev/daik…
ami-GS Mar 15, 2024
1dd8e52
temporal fix
ami-GS Mar 15, 2024
23b7b06
Merge branch 'main' into dev/daiki/xdp_split_functional
ami-GS Mar 18, 2024
b3743dc
make XDP optional
ami-GS Mar 19, 2024
f34a3e8
use ShutdownSqe
ami-GS Mar 19, 2024
560730b
fix yaml files
ami-GS Mar 20, 2024
8c8676f
remove gcc-multilib and include explicitly
ami-GS Mar 21, 2024
d69e762
fix tcp socket to allocate raw
ami-GS Mar 21, 2024
943d91f
Merge branch 'main' into dev/daiki/xdp_split_functional
ami-GS Mar 22, 2024
649b948
install xdp deps for linux
ami-GS Mar 22, 2024
d7a50b4
Merge branch 'main' into dev/daiki/xdp_split_functional
ami-GS Mar 22, 2024
a2b332e
libxdp bug workaround
ami-GS Mar 23, 2024
baf452e
oops
ami-GS Mar 23, 2024
1ff6ba3
fix idempotency and add one more dep
ami-GS Mar 23, 2024
29f9898
TODO: remove this
ami-GS Mar 23, 2024
eab2f7f
enable x86_64 and Ubuntu22.04
ami-GS Mar 24, 2024
1802e4e
fix type mismatch of uint16_t and UINT16
ami-GS Mar 25, 2024
b4c7ce6
add aarch64
ami-GS Mar 25, 2024
e375786
fix
ami-GS Mar 25, 2024
35b63f1
use flag
ami-GS Mar 25, 2024
c48bdbb
change arch check and file copy
ami-GS Mar 25, 2024
92a0b8c
fix static build and android
ami-GS Mar 25, 2024
c074126
update clog
ami-GS Mar 25, 2024
05d86a8
clog fix
ami-GS Mar 25, 2024
7999808
fix Clang link issue
ami-GS Mar 25, 2024
063f325
fix CodeCheck and prepare-machine for build
ami-GS Mar 25, 2024
bdd0161
disable ANDROID for xdp
ami-GS Mar 26, 2024
cdaf797
fix clog
ami-GS Mar 26, 2024
d1639dd
Support XDP from 'test.ps1 -UseXdp'
ami-GS Mar 26, 2024
dd6a74d
link libxdp from artifact
ami-GS Mar 26, 2024
5acb167
fix dependency
ami-GS Mar 26, 2024
671e112
fix dep
ami-GS Mar 26, 2024
1d69ad9
update xdp-tools to v1.4.2
ami-GS Mar 26, 2024
ebb21b9
fix runtime deps
ami-GS Mar 26, 2024
7106913
fix deps
ami-GS Mar 27, 2024
8fa7f5c
Fix Umem addressing and make Rx path async to efficiently fill FQ
ami-GS Mar 27, 2024
2983028
fix CodeQL analyze deps
ami-GS Mar 28, 2024
e3be961
fix Analyze
ami-GS Mar 28, 2024
7f68f6e
fix analyze
ami-GS Mar 28, 2024
d3c7eb6
linux xdp to use sudo
ami-GS Mar 28, 2024
2100f5e
change 'if' order
ami-GS Mar 28, 2024
360c28a
path fix?
ami-GS Mar 28, 2024
d434b3d
sudo pwsh
ami-GS Mar 28, 2024
89a361c
add retry for xsk_socket__create when returns -EBUSY
ami-GS Mar 29, 2024
aa8cf57
cleanup xdp program before running
ami-GS Mar 29, 2024
70d4622
Fix cargo build --all
ami-GS Mar 29, 2024
5d9b1e8
fix cargo test dynamic linking
ami-GS Mar 31, 2024
9490d94
remove Sanitize for initial commit
ami-GS Mar 31, 2024
d17770c
generate artifact
ami-GS Mar 31, 2024
e30e083
fix yml syntax issue
ami-GS Apr 1, 2024
19f7abc
fix CodeQl Analysys
ami-GS Apr 1, 2024
3cae069
forget to add Partition declaration
ami-GS Apr 1, 2024
e694653
Use Rundown to wait for deleting socket before binding next
ami-GS Apr 2, 2024
2c41090
fix Rundown for all xdp socket
ami-GS Apr 2, 2024
50844d2
Fix EBUSY issue when binding xdp socket
ami-GS Apr 3, 2024
c9fd1e3
explictly disable port sharing test
ami-GS Apr 3, 2024
90f81b6
refactoring and cleanup
ami-GS Apr 4, 2024
d64dbd1
fix pipeline. else for normal linux socket
ami-GS Apr 4, 2024
02ffcca
Fix cargo linking issue?
ami-GS Apr 4, 2024
ed95167
Merge branch 'main' into dev/daiki/xdp_split_functional
ami-GS Apr 4, 2024
10301b3
disable xdp for cargo build
ami-GS Apr 5, 2024
06acce7
fix codeql
ami-GS Apr 5, 2024
2d17796
Merge branch 'main' into dev/daiki/xdp_split_functional
ami-GS Apr 5, 2024
d8d122b
rollback build.rs
ami-GS Apr 5, 2024
4c878e0
fix package runtime link and codeql
ami-GS Apr 5, 2024
faedae1
wrap packet dump function to DEBUG macro
ami-GS Apr 5, 2024
b1633e6
check dependency
ami-GS Apr 5, 2024
0bf636f
another check
ami-GS Apr 5, 2024
293f8bd
leave only LD_LIBRARY_PATH
ami-GS Apr 5, 2024
6b49bb6
iterate attach mode
ami-GS Apr 5, 2024
8aa3402
fix codeql
ami-GS Apr 5, 2024
d56ce17
add IP address check in XDP prog
ami-GS Apr 8, 2024
8c8f463
break is enough & avoid compiler bug
ami-GS Apr 8, 2024
c305f11
fix CodeCheck
ami-GS Apr 8, 2024
0222e70
Fix use of RxHeadroom
ami-GS Apr 8, 2024
606c3b8
fix umem leak when failed to initialize
ami-GS Apr 8, 2024
0c782c8
use pool alloc for Interface
ami-GS Apr 8, 2024
51cdebc
fix ipv4 check
ami-GS Apr 8, 2024
d3dc703
remove workaround
ami-GS Apr 9, 2024
b4decc6
cleanup. from snake case to pascal case
ami-GS Apr 9, 2024
d1588b6
fix clog
ami-GS Apr 9, 2024
6cd580c
try ubuntu20.04 and enable stress test
ami-GS Apr 9, 2024
706e4da
few fix of stress test yml
ami-GS Apr 10, 2024
56a7fb1
integrate package libxdp linking
ami-GS Apr 10, 2024
600ad66
fix RX parse error case
ami-GS Apr 10, 2024
ca7bde5
disable ubuntu20.04
ami-GS Apr 12, 2024
4569402
remove 2 env variable to run
ami-GS Apr 13, 2024
23a9291
add document
ami-GS Apr 13, 2024
f96feab
fix ForBuild && clog
ami-GS Apr 13, 2024
4b829ce
stop installing deps on Ubuntu 20.04
ami-GS Apr 13, 2024
7ce0f8d
Merge branch 'main' into dev/daiki/xdp_split_functional
ami-GS Apr 15, 2024
31d817f
fix build
ami-GS Apr 16, 2024
1d43c09
fix libc6-dev-i386 side effect
ami-GS Apr 16, 2024
ae95b04
Add one more dependency (libzstd) for static build
ami-GS Apr 16, 2024
e27243f
new line fix
ami-GS Apr 16, 2024
05376dc
always install libc6-dev-i386
ami-GS Apr 16, 2024
6c792c8
fix install order
ami-GS Apr 16, 2024
561044e
fix CMakeLists.txt to cover qns.Dockerfile case
ami-GS Apr 16, 2024
2f4f171
remove env var dep, update doc
ami-GS Apr 16, 2024
c88fe20
fix stress.yml
ami-GS Apr 16, 2024
7f6fbd0
add env var for xdp knob and fix stress
ami-GS Apr 17, 2024
2b25b39
fix stress
ami-GS Apr 17, 2024
d31a0f5
update clog
ami-GS Apr 17, 2024
57bfa9f
update doc
ami-GS Apr 17, 2024
714a655
Fix doc appearance and words
ami-GS Apr 17, 2024
6768679
add UseXdp
ami-GS Apr 17, 2024
f0dab8b
Update docs/BUILD.md
ami-GS Apr 19, 2024
bbe0a5a
Update docs/BUILD.md
ami-GS Apr 19, 2024
ad74b33
use SKB mode for now
ami-GS Apr 19, 2024
af18bef
move Linux XDP logging to TSG.md
ami-GS Apr 19, 2024
9a7acdf
stop warning print from libxdp/libbpf
ami-GS Apr 19, 2024
14a8fd7
move CxPlat definitions to each plat
ami-GS Apr 19, 2024
e961a3b
initial commit
ami-GS Apr 21, 2024
8ecc440
common xdp utilities
ami-GS Apr 21, 2024
5f1da03
move sudo inside scripts
ami-GS Apr 22, 2024
6d2339c
fix comments
ami-GS Apr 22, 2024
037ea20
Merge branch 'dev/daiki/xdp_merge' into dev/daiki/xdp_split_functional
ami-GS Apr 22, 2024
c87e720
fix clog
ami-GS Apr 22, 2024
ed69f0a
Merge branch 'main' into dev/daiki/xdp_split_functional
ami-GS Apr 22, 2024
e560f3f
fix ifdef syntax
ami-GS Apr 22, 2024
2cfa516
fix annotations
ami-GS Apr 22, 2024
9cd42d0
all arch to build dataptah_xplat.c
ami-GS Apr 22, 2024
17825c0
fix mac build
ami-GS Apr 22, 2024
74eed9f
fix lisence of ebpf prog
ami-GS Apr 22, 2024
eebc505
fix spin.ps1 to propagate parameter
ami-GS Apr 22, 2024
4eda469
Ether and IPv6 was not set appropriately
ami-GS Apr 24, 2024
8924bed
Update docs/TSG.md
nibanks Apr 25, 2024
1703841
remove warning supression
ami-GS Apr 25, 2024
98a1222
remove BuildLibXdpFromSource
ami-GS Apr 25, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .github/workflows/build-reuse-unix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,10 @@ on:
required: false
default: ''
type: string
xdp:
required: false
default: ''
type: string
sanitize:
required: false
default: ''
Expand Down Expand Up @@ -95,7 +99,7 @@ jobs:
chown -R $(id -u):$(id -g) $PWD
- name: Prepare Machine
shell: pwsh
run: scripts/prepare-machine.ps1 ${{ inputs.plat == 'linux' && '-ForContainerBuild' || '-ForBuild' }} -Tls ${{ inputs.tls }}
run: scripts/prepare-machine.ps1 ${{ inputs.plat == 'linux' && '-ForContainerBuild' || '-ForBuild' }} -Tls ${{ inputs.tls }} ${{ inputs.xdp }}
- name: Build For Test
if: inputs.build == '-Test'
shell: pwsh
Expand Down
3 changes: 3 additions & 0 deletions .github/workflows/codeql-analysis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,11 @@ jobs:
# Install dependencies and build submodules before starting analysis.
- run: |
sudo apt-add-repository ppa:lttng/stable-2.13
sudo apt-add-repository "deb http://mirrors.kernel.org/ubuntu noble main" -y
sudo apt-get update
sudo apt-get install -y liblttng-ust-dev libnuma-dev
sudo apt-get install -y libxdp-dev libbpf-dev libnl-3-dev libnl-genl-3-dev libnl-route-3-dev zlib1g-dev zlib1g pkg-config m4 clang libpcap-dev libelf-dev
sudo apt-get install -y --no-install-recommends libc6-dev-i386
sudo sh scripts/install-powershell-docker.sh
mkdir build
cd build
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/stress.yml
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ jobs:
{ config: "Debug", plat: "macos", os: "macos-12", arch: "x64", tls: "openssl3", build: "-Test" },
{ config: "Debug", plat: "linux", os: "ubuntu-20.04", arch: "x64", tls: "openssl", sanitize: "-Sanitize", build: "-Test" },
{ config: "Debug", plat: "linux", os: "ubuntu-20.04", arch: "x64", tls: "openssl3", sanitize: "-Sanitize", build: "-Test" },
{ config: "Debug", plat: "linux", os: "ubuntu-22.04", arch: "x64", tls: "openssl3", build: "-Test" },
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

XDP doesn't work with sanitizer? Why?

Copy link
Contributor Author

@ami-GS ami-GS Apr 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I previously saw data alignment related error by test.yml.
Not sure stress test. I enabled stress test later than test.yml
I disabled sanitizer from functionality point of view

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should follow up (in a later PR) on the sanitizer.

]
uses: ./.github/workflows/build-reuse-unix.yml
with:
Expand All @@ -72,6 +73,7 @@ jobs:
vec: [
{ config: "Debug", plat: "linux", os: "ubuntu-20.04", arch: "x64", tls: "openssl", sanitize: "-Sanitize", build: "-Test" },
{ config: "Debug", plat: "linux", os: "ubuntu-20.04", arch: "x64", tls: "openssl3", sanitize: "-Sanitize", build: "-Test" },
{ config: "Debug", plat: "linux", os: "ubuntu-22.04", arch: "x64", tls: "openssl3", xdp: "-UseXdp", build: "-Test" },
{ config: "Debug", plat: "macos", os: "macos-12", arch: "x64", tls: "openssl", build: "-Test" },
{ config: "Debug", plat: "macos", os: "macos-12", arch: "x64", tls: "openssl3", build: "-Test" },
{ config: "Debug", plat: "windows", os: "windows-2022", arch: "x64", tls: "schannel", sanitize: "-Sanitize", build: "-Test" },
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ jobs:
{ config: "Debug", plat: "linux", os: "ubuntu-20.04", arch: "x64", tls: "openssl3", sanitize: "-Sanitize", build: "-Test" },
{ config: "Debug", plat: "linux", os: "ubuntu-22.04", arch: "x64", tls: "openssl", sanitize: "-Sanitize", build: "-Test" },
{ config: "Debug", plat: "linux", os: "ubuntu-22.04", arch: "x64", tls: "openssl3", sanitize: "-Sanitize", build: "-Test" },
{ config: "Debug", plat: "linux", os: "ubuntu-22.04", arch: "x64", tls: "openssl3", build: "-Test" },
{ config: "Debug", plat: "linux", os: "ubuntu-22.04", arch: "x64", tls: "openssl3", systemcrypto: "-UseSystemOpenSSLCrypto", sanitize: "-Sanitize", build: "-Test" },
]
uses: ./.github/workflows/build-reuse-unix.yml
Expand All @@ -96,6 +97,7 @@ jobs:
{ config: "Debug", plat: "linux", os: "ubuntu-20.04", arch: "x64", tls: "openssl3", sanitize: "-Sanitize", build: "-Test" },
{ config: "Debug", plat: "linux", os: "ubuntu-22.04", arch: "x64", tls: "openssl", sanitize: "-Sanitize", build: "-Test" },
{ config: "Debug", plat: "linux", os: "ubuntu-22.04", arch: "x64", tls: "openssl3", sanitize: "-Sanitize", build: "-Test" },
{ config: "Debug", plat: "linux", os: "ubuntu-22.04", arch: "x64", tls: "openssl3", xdp: "-UseXdp", build: "-Test" },
{ config: "Debug", plat: "linux", os: "ubuntu-22.04", arch: "x64", tls: "openssl3", systemcrypto: "-UseSystemOpenSSLCrypto", sanitize: "-Sanitize", build: "-Test" },
{ config: "Debug", plat: "windows", os: "windows-2019", arch: "x64", tls: "openssl", build: "-Test" },
{ config: "Debug", plat: "windows", os: "windows-2019", arch: "x64", tls: "openssl3", build: "-Test" },
Expand Down
15 changes: 9 additions & 6 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,9 @@ elseif (APPLE)
set(CX_PLATFORM "darwin")
elseif (UNIX)
set(CX_PLATFORM "linux")
file(STRINGS "/etc/lsb-release" LSB_RELEASE_CONTENT)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line broke our Alpine docker build, using /etc/os-release would be better as it is present in both distributions.

string(REGEX MATCH "DISTRIB_RELEASE=([0-9]+\\.[0-9]+)" _ ${LSB_RELEASE_CONTENT})
set(UBUNTU_VERSION ${CMAKE_MATCH_1})
endif()
message(STATUS "QUIC Platform: ${CX_PLATFORM}")

Expand Down Expand Up @@ -434,13 +437,13 @@ if (NOT MSVC AND NOT APPLE AND NOT ANDROID)
endif()
endif()

if(WIN32)
if (CMAKE_GENERATOR_PLATFORM STREQUAL "")
string(TOLOWER ${CMAKE_SYSTEM_PROCESSOR} SYSTEM_PROCESSOR)
else()
string(TOLOWER ${CMAKE_GENERATOR_PLATFORM} SYSTEM_PROCESSOR)
endif()
if (CMAKE_GENERATOR_PLATFORM STREQUAL "")
string(TOLOWER ${CMAKE_SYSTEM_PROCESSOR} SYSTEM_PROCESSOR)
else()
string(TOLOWER ${CMAKE_GENERATOR_PLATFORM} SYSTEM_PROCESSOR)
endif()

if(WIN32)
# Generate the MsQuicEtw header file.
file(MAKE_DIRECTORY ${QUIC_BUILD_DIR}/inc)

Expand Down
53 changes: 53 additions & 0 deletions docs/BUILD.md
Original file line number Diff line number Diff line change
Expand Up @@ -181,6 +181,59 @@ sudo dnf install openssl-devel
sudo dnf install libatomic
```

#### Linux XDP
Linux XDP is experimentally supported on amd64 && Ubuntu 22.04LTS.
Commands below automatically install dependencies and setup runtime environment.
```sh
pwsh ./scripts/prepare-machine.ps1 -UseXdp
pwsh ./scripts/build.ps1
export MSQUIC_ENABLE_XDP=1
```

`./scripts/prepare-machine.ps1` internally does the below commands:
```sh
# for libxdp v1.4.2
sudo apt-add-repository "deb http://mirrors.kernel.org/ubuntu noble main" -y

# install runtime dependencies
sudo apt-get install -y libxdp1 libbpf1 libnl-3-200 libnl-route-3-200 libnl-genl-3-200

# install build dependencies
sudo apt-get --no-install-recommends -y install libxdp-dev libbpf-dev libnl-3-dev libnl-genl-3-dev libnl-route-3-dev zlib1g-dev zlib1g pkg-config m4 clang libpcap-dev libelf-dev libc6-dev-i386

# Optional. This is required when you run test with duonic (XDP capable virtual nic pair)
sudo apt-get -y install iproute2 iptables
sudo ./scripts/duonic.sh install
```

Test
```sh
# "sudo" and MSQUIC_ENABLE_XDP=1 required
# You can explicitly specify directory of datapath_raw_xdp_kern.o by MSQUIC_XDP_OBJECT_PATH
# By default, libmsquic.so searchs for same directory as its executable
# If something failed, fallback to normal socket
sudo MSQUIC_ENABLE_XDP=1 ./artifacts/bin/linux/x64_Debug_openssl3/msquictest --duoNic
```

**Q&A**
- Q: Is this workload really running on XDP?
A: If you have the `xdp-dump` command, try using `sudo xdp-dump --list-interfaces`. The `xdp_main` function is located in `src/platform/datapath_raw_xdp_linux_kern.c`. If none of the interfaces load the XDP program, something must be wrong.
```
$ sudo ./xdp-tools/xdp-dump/xdpdump --list-interfaces
Interface Prio Program name Mode ID Tag Chain actions
--------------------------------------------------------------------------------------
lo <No XDP program loaded!>
eth0 <No XDP program loaded!>
docker0 <No XDP program loaded!>
duo2 xdp_dispatcher native 608211 4d7e87c0d30db711
=> 50 xdp_main 608220 c8fcabdd9e3895f3 XDP_PASS
duo1 xdp_dispatcher native 608225 4d7e87c0d30db711
=> 50 xdp_main 608228 c8fcabdd9e3895f3 XDP_PASS
```

- Q: Is Ubuntu 20.04LTS supported?
A: Not officially, but you can still **build** it by running `apt-get upgrade linux-libc-dev`. Please be aware of potential side effects from the **upgrade**.

### macOS
The build needs CMake and compiler.

Expand Down
20 changes: 20 additions & 0 deletions docs/TSG.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,26 @@ This document is meant to be a step-by-step guide for trouble shooting any issue

See [Tracing](./Diagnostics.md#Built-in-Tracing)

### Linux XDP logging?
For XDP layer, enable `DEBUG` flag in `src/platform/CMakeLists.txt`.
You can see incomming packets information by `sudo cat /sys/kernel/debug/tracing/trace_pipe`.
**Your workload must become too slow.**
```
msquictest-3797496 [005] ..s1. 2079546.776875: bpf_trace_printk: ========> To ifacename : [duo2], RxQueueID:0
msquictest-3797496 [005] ..s1. 2079546.776875: bpf_trace_printk: Eth[244] SRC: 00:00:00:00:00:00 => DST:22:22:22:22:00:02
msquictest-3797496 [005] ..s1. 2079546.776876: bpf_trace_printk: Ipv4 TotalLen:[230] Src: 192.168.1.11 => Dst: 192.168.1.12
msquictest-3797496 [005] ..s1. 2079546.776877: bpf_trace_printk: UDP[202]: SRC: 43829 DST:58141
msquictest-3797496 [005] ..s1. 2079546.776877: bpf_trace_printk: [ec 00 00 00 01 00 09 c0 30 3d 49 a2]
msquictest-3797496 [005] ..s1. 2079546.776878: bpf_trace_printk: Redirect to QUIC service. IpMatch:1, PortMatch:1, SocketExists:1, Redirection:4

msquictest-3797496 [005] ..s1. 2079546.777235: bpf_trace_printk: ========> To ifacename : [duo1], RxQueueID:0
msquictest-3797496 [005] ..s1. 2079546.777310: bpf_trace_printk: Eth[1262] SRC: 00:00:00:00:00:00 => DST:22:22:22:22:00:01
msquictest-3797496 [005] ..s1. 2079546.777323: bpf_trace_printk: Ipv4 TotalLen:[1248] Src: 192.168.1.12 => Dst: 192.168.1.11
msquictest-3797496 [005] ..s1. 2079546.777323: bpf_trace_printk: UDP[1220]: SRC: 58141 DST:43829
msquictest-3797496 [005] ..s1. 2079546.777324: bpf_trace_printk: [c0 00 00 00 01 09 c0 30 3d 49 a2 56]
msquictest-3797496 [005] ..s1. 2079546.777325: bpf_trace_printk: Redirect to QUIC service. IpMatch:1, PortMatch:1, SocketExists:1, Redirection:4
```

## Understanding Error Codes

Some error codes are MsQuic specific (`QUIC_STATUS_*`), and some are simply a passthrough from the platform. You can find the MsQuic specific error codes in the platform specific header ([msquic_posix.h](../src/inc/msquic_posix.h), [msquic_winkernel.h](../src/inc/msquic_winkernel.h), or [msquic_winuser.h](../src/inc/msquic_winuser.h)).
Expand Down
3 changes: 2 additions & 1 deletion scripts/build.rs
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@ fn main() {
config
.define("QUIC_ENABLE_LOGGING", logging_enabled)
.define("QUIC_TLS", "openssl")
.define("QUIC_OUTPUT_DIR", "../lib");
.define("QUIC_OUTPUT_DIR", "../lib")
.define("QUIC_CARGO_BUILD", "on");

match target.as_str() {
"x86_64-apple-darwin" => config
Expand Down
3 changes: 3 additions & 0 deletions scripts/clog.inputs
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@
../src/platform/hashtable.c
../src/platform/datapath_winuser.c
../src/platform/datapath_linux.c
../src/platform/datapath_unix.c
../src/platform/datapath_xplat.c
../src/platform/datapath_raw_dpdk.c
../src/platform/datapath_raw_socket.c
../src/platform/datapath_raw_socket_win.c
Expand All @@ -22,6 +24,7 @@
../src/platform/datapath_raw_xdp_linux.c
../src/platform/datapath_raw_win.c
../src/platform/datapath_raw_linux.c
../src/platform/datapath_raw.c
../src/platform/crypt_bcrypt.c
../src/platform/platform_winuser.c
../src/platform/toeplitz.c
Expand Down
102 changes: 102 additions & 0 deletions scripts/duonic.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
#!/bin/bash

# Set the number of NIC pairs
NumNicPairs=1

if [ "$1" == "install" ]; then
# Configure each pair separately with its own hard-coded subnet, ie 192.168.x.0/24 and fc00::x/112
for ((i=1; i<=NumNicPairs; i++)); do
echo "Plumbing IP config for pair $i"

# Generate the "ID" of the NICs, eg 1 and 2 for the first pair
nic1="duo$((i * 2 - 1))"
nic2="duo$((i * 2))"

# Create veth pair
sudo ip link add ${nic1} type veth peer name ${nic2}

# Set the veth interfaces up
sudo ip link set ${nic1} up
sudo ip link set ${nic2} up

# Assign IPv4 addresses
sudo ip addr add 192.168.${i}.11/24 dev ${nic1}
sudo ip addr add 192.168.${i}.12/24 dev ${nic2}

# Assign IPv6 addresses
sudo ip -6 addr add fc00::${i}:11/112 dev ${nic1}
sudo ip -6 addr add fc00::${i}:12/112 dev ${nic2}

# Add static neighbor entries (ARP)
sudo ip neigh add 192.168.${i}.12 lladdr 22:22:22:22:00:0$((i * 2)) dev ${nic1} nud permanent
sudo ip neigh add 192.168.${i}.11 lladdr 22:22:22:22:00:0$((i * 2 - 1)) dev ${nic2} nud permanent

# Add static neighbor entries (IPv6)
sudo ip -6 neigh add fc00::${i}:12 lladdr 22:22:22:22:00:0$((i * 2)) dev ${nic1} nud permanent
sudo ip -6 neigh add fc00::${i}:11 lladdr 22:22:22:22:00:0$((i * 2 - 1)) dev ${nic2} nud permanent

# Configure routing rules for IPv4
sudo ip route add 192.168.${i}.12/32 dev ${nic1} metric 0
sudo ip route add 192.168.${i}.11/32 dev ${nic2} metric 0



# Configure routing rules for IPv6
sudo ip -6 route add fc00::${i}:12/128 dev ${nic1} metric 0
sudo ip -6 route add fc00::${i}:11/128 dev ${nic2} metric 0

# Configure firewall rules for IPv4
sudo iptables -A INPUT -p all -s 192.168.${i}.0/24 -i ${nic1} -j ACCEPT
sudo iptables -A INPUT -p all -s 192.168.${i}.0/24 -i ${nic2} -j ACCEPT

# Configure firewall rules for IPv6
sudo ip6tables -A INPUT -p all -s fc00::${i}:0/112 -i ${nic1} -j ACCEPT
sudo ip6tables -A INPUT -p all -s fc00::${i}:0/112 -i ${nic2} -j ACCEPT

sleep 2
sudo ip route change 192.168.${i}.12 dev duo1 src 192.168.${i}.11
sudo ip route change 192.168.${i}.11 dev duo2 src 192.168.${i}.12

sudo ip -6 route change fc00::${i}:12 dev duo1 src fc00::${i}:11
sudo ip -6 route change fc00::${i}:11 dev duo2 src fc00::${i}:12
done
elif [ "$1" == "uninstall" ]; then
# Cleanup each pair separately
for ((i=1; i<=NumNicPairs; i++)); do
echo "Cleaning up pair $i"

# Generate the "ID" of the NICs, eg 1 and 2 for the first pair
nic1="duo$((i * 2 - 1))"
nic2="duo$((i * 2))"

# Delete firewall rules for IPv4
sudo iptables -D INPUT -p all -s 192.168.${i}.0/24 -i ${nic1} -j ACCEPT
sudo iptables -D INPUT -p all -s 192.168.${i}.0/24 -i ${nic2} -j ACCEPT

# Delete firewall rules for IPv6
sudo ip6tables -D INPUT -p all -s fc00::${i}:0/112 -i ${nic1} -j ACCEPT
sudo ip6tables -D INPUT -p all -s fc00::${i}:0/112 -i ${nic2} -j ACCEPT

# Remove routing rules for IPv4
sudo ip route del 192.168.${i}.12/32 dev ${nic1}
sudo ip route del 192.168.${i}.11/32 dev ${nic2}

# Remove routing rules for IPv6
sudo ip -6 route del fc00::${i}:12/128 dev ${nic1}
sudo ip -6 route del fc00::${i}:11/128 dev ${nic2}

# Remove static neighbor entries (ARP)
sudo ip neigh del 192.168.${i}.12 lladdr 22:22:22:22:00:0$((i * 2)) dev ${nic1}
sudo ip neigh del 192.168.${i}.11 lladdr 22:22:22:22:00:0$((i * 2 - 1)) dev ${nic2}

# Remove static neighbor entries (IPv6)
sudo ip -6 neigh del fc00::${i}:12 lladdr 22:22:22:22:00:0$((i * 2)) dev ${nic1}
sudo ip -6 neigh del fc00::${i}:11 lladdr 22:22:22:22:00:0$((i * 2 - 1)) dev ${nic2}

# Remove the veth pair
sudo ip link delete ${nic1}
done
else
echo "Usage: $0 {install|uninsatll}"
exit 1
fi
36 changes: 30 additions & 6 deletions scripts/prepare-machine.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -228,13 +228,18 @@ function Uninstall-Xdp {

# Installs DuoNic from the CoreNet-CI repo.
function Install-DuoNic {
if (!$IsWindows) { return } # Windows only
# Install the DuoNic driver.
Write-Host "Installing DuoNic driver"
$DuoNicPath = Join-Path $SetupPath duonic
$DuoNicScript = (Join-Path $DuoNicPath duonic.ps1)
if (!(Test-Path $DuoNicScript)) { Write-Error "Missing file: $DuoNicScript" }
Invoke-Expression "cmd /c `"pushd $DuoNicPath && pwsh duonic.ps1 -Install`""
if ($IsWindows) {
Write-Host "Installing DuoNic driver"
$DuoNicPath = Join-Path $SetupPath duonic
$DuoNicScript = (Join-Path $DuoNicPath duonic.ps1)
if (!(Test-Path $DuoNicScript)) { Write-Error "Missing file: $DuoNicScript" }
Invoke-Expression "cmd /c `"pushd $DuoNicPath && pwsh duonic.ps1 -Install`""
} elseif ($IsLinux) {
Write-Host "Creating DuoNic endpoints"
$DuoNicScript = Join-Path $PSScriptRoot "duonic.sh"
Invoke-Expression "sudo bash $DuoNicScript install"
}
}

function Update-Path($NewPath) {
Expand Down Expand Up @@ -504,6 +509,15 @@ if ($IsLinux) {
sudo apt-get install -y ruby ruby-dev rpm
sudo gem install public_suffix -v 4.0.7
sudo gem install fpm

# XDP dependencies
if ((bash -c 'lsb_release -r') -match '22.04') {
sudo apt-get -y install --no-install-recommends libc6-dev-i386 # for building xdp programs
sudo apt-add-repository "deb http://mirrors.kernel.org/ubuntu noble main" -y
sudo apt-get update -y
sudo apt-get -y install libxdp-dev libbpf-dev
sudo apt-get -y install libnl-3-dev libnl-genl-3-dev libnl-route-3-dev zlib1g-dev zlib1g pkg-config m4 clang libpcap-dev libelf-dev
}
}

if ($ForTest) {
Expand All @@ -512,6 +526,16 @@ if ($IsLinux) {
sudo apt-get install -y lttng-tools
sudo apt-get install -y liblttng-ust-dev
sudo apt-get install -y gdb
if ((bash -c 'lsb_release -r') -match '22.04') {
sudo apt-add-repository "deb http://mirrors.kernel.org/ubuntu noble main" -y
sudo apt-get update -y
sudo apt-get install -y libxdp1 libbpf1
sudo apt-get install -y libnl-3-200 libnl-route-3-200 libnl-genl-3-200
if ($UseXdp) {
sudo apt-get -y install iproute2 iptables
Install-DuoNic
}
}

# Enable core dumps for the system.
Write-Host "Setting core dump size limit"
Expand Down
6 changes: 5 additions & 1 deletion scripts/spin.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -187,4 +187,8 @@ if (![string]::IsNullOrWhiteSpace($ExtraArtifactDir)) {
}

# Run the script.
Invoke-Expression ($RunExecutable + " " + $Arguments)
if ($IsLinux -and $UseXdp) {
Invoke-Expression ('sudo MSQUIC_ENABLE_XDP=1 pwsh -c "$RunExecutable $Arguments"')
} else {
Invoke-Expression ($RunExecutable + " " + $Arguments)
}
Loading
Loading