-
Notifications
You must be signed in to change notification settings - Fork 39
/
README
325 lines (241 loc) · 11.9 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
SPDX-License-Identifier: BSD-3-Clause
Copyright (c) 2018-2019 Linaro Limited
Copyright (c) 2019-2024 Nokia
1. Rationale
=================================================
This is an effort to port ODP on top of DPDK to accelerate packet processing on
systems equipped with NIC's that supports DPDK.
Prerequisites and considerations:
--------------------------------
- 8GB of RAM recommended, 4 might be enough too
- it's recommended to obtain a DPDK supported network card (many come in dual
port configuration, mostly dedicated to server usage)
- it's also possible to use odp-dpdk for evaluation purposes without a DPDK
compatible NIC, using the pcap poll mode driver
- DPDK code must be downloaded, configured and compiled, details below
- ODP-DPDK has been compiled and tested on an x86 host with Ubuntu 22.04 LTS
(5.15.0 kernel).
- DPDK only works on a selected range of network cards. The list of known and
supported devices can be found in the DPDK documentation:
https://doc.dpdk.org/guides/nics/index.html
2. Preparing DPDK
=================================================
Please refer to https://dpdk.org/doc/guides/linux_gsg/build_dpdk.html for more
details on how to build DPDK. Best effort is done to provide some help on DPDK
cmds below for Ubuntu, where it has been compiled and tested.
On Ubuntu install pcap development library:
sudo apt-get install libpcap-dev
Right now ODP-DPDK supports DPDK v21.11, v22.11 (recommended version), and
v23.11.
Compile DPDK
------------
Fetch the DPDK code:
git clone https://dpdk.org/git/dpdk-stable --branch 22.11 --depth 1 ./<dpdk-dir>
Prepare the build directory:
cd <dpdk-dir>
meson build
cd build
Optionally, configure the location where DPDK will be installed. By default,
DPDK will be installed in /usr/local:
meson configure -Dprefix=$(pwd)/../install
Build and install DPDK:
ninja install
3. Compile ODP-DPDK
=================================================
Build dependencies are listed in DEPENDENCIES. Use absolute DPDK directory
path with the --with-dpdk-path option.
cd <odp-dir>
./bootstrap
The following step depends on whether ODP shared libraries are to be built.
SHARED libraries:
./configure --enable-dpdk-shared
STATIC libraries (better performance):
./configure --disable-shared
Or, if DPDK was not installed to the default location, set PKG_CONFIG_PATH:
PKG_CONFIG_PATH=<dpdk-dir>/install/lib/x86_64-linux-gnu/pkgconfig ./configure
Once configure has completed successfully:
make
4. Prepare DPDK for running ODP-DPDK examples
=================================================
Reserve hugepages:
-----------------
To reserve huge pages, which is needed for DPDK, execute following commands
(these are usually needed only once after the system has started):
sudo sh -c 'echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages'
If you are running on a multi-node machine then hugepages have to be reserved on
each node:
ls /sys/devices/system/node
sudo sh -c 'echo 1024 > /sys/devices/system/node/node*/hugepages/hugepages-2048kB/nr_hugepages'
Mount hugetlbfs:
---------------
sudo mkdir /mnt/huge
sudo mount -t hugetlbfs nodev /mnt/huge
Insert DPDK kernel module:
-------------------------
DPDK uses userspace poll mode drivers, so it's necessary to insert a couple of
modules to allow DPDK to map the NIC's registers to userspace:
If UIO is used:
sudo modprobe uio
cd <dpdk-dir>
sudo insmod x86_64-native-linuxapp-gcc/kmod/igb_uio.ko
or VFIO is used (requires kernel and BIOS support e.g. iommu=pt intel_iommu=on)
sudo modprobe vfio-pci
Bind NIC's to DPDK:
------------------
The DPDK code contains a tool used to bind drivers to the network cards.
cd <dpdk-dir>
./usertools/dpdk-devbind.py --status
This command produces output that is similar to the one given below:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Network devices using DPDK-compatible driver
============================================
0000:05:00.0 'Ethernet 10G 2P X520 Adapter' drv=igb_uio unused=
0000:05:00.1 'Ethernet 10G 2P X520 Adapter' drv=igb_uio unused=
Network devices using kernel driver
===================================
0000:01:00.0 'NetXtreme II BCM5709 Gigabit Ethernet' if=eth0 drv=bnx2 unused=<none> *Active*
0000:01:00.1 'NetXtreme II BCM5709 Gigabit Ethernet' if=eth1 drv=bnx2 unused=<none>
0000:07:00.0 'T320 10GbE Dual Port Adapter' if=eth2,eth3 drv=cxgb3 unused=<none>
Other network devices
=====================
<none>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Bind using interface name:
-------------------------
The easiest way is to let the tool automatically switch the regular drivers. For
that the interface must not be active i.e. no IP addresses assigned:
sudo ifconfig eth0 0
sudo ifconfig eth1 0
If UIO is used:
sudo ./usertools/dpdk-devbind.py --bind=igb_uio eth0
sudo ./usertools/dpdk-devbind.py --bind=igb_uio eth1
or if VFIO is used:
sudo ./usertools/dpdk-devbind.py --bind=vfio-pci eth0
sudo ./usertools/dpdk-devbind.py --bind=vfio-pci eth1
Bind using PCI ids:
------------------
Another way is to remove the regular drivers and use PCI ids:
sudo rmmod ixgbe
sudo ./usertools/dpdk-devbind.py --bind=igb_uio 05:00.0
sudo ./usertools/dpdk-devbind.py --bind=igb_uio 05:00.1
Unbind network cards from DPDK:
------------------------------
To restore the NIC's back to kernel use something like this:
sudo ./usertools/dpdk-devbind.py --bind=ixgbe 05:00.0
sudo ./usertools/dpdk-devbind.py --bind=ixgbe 05:00.1
5. Running ODP apps
=================================================
ODP-DPDK applications need to be run as root. You may also need to
supply DPDK command line parameters either as a null-terminated array of
char's to odp_global_init()'s platform_params parameter:
odp_global_init([params], "--no-huge");
Or, if it's NULL the platform tries to read the ODP_PLATFORM_PARAMS environment
variable.
The coremask (-c) is calculated by ODP-DPDK based on the process affinity at
startup. You can influence that with 'taskset'. DPDK init changes the affinity
of the calling thread, so after it returns the original affinity is restored.
Only the first active core is passed to rte_eal_init(), as the rest would be
used for DPDK's special lcore threads, which are only available through
rte_eal_[mp_]remote_launch(), but not through ODP API's. Nevertheless,
odp_local_init() makes sure for the rest of the DPDK libraries ODP threads look
like proper DPDK threads.
Example how to run an ODP-DPDK L2 forwarding application:
sudo ./odp_l2fwd -i 0,1 -c 2
-i 0,1 - interface numbers
-c 2 - number of worker cpus
6. Howto debug DPDK apps on the host
=================================================
For example you need to debug some l2fwd application. Then network configuration
might be:
<veth1-2> <-----> <veth2-1> (iface0 DPDK L2FWD APP iface1) <veth2-3> <-----> <veth3-2>
Where:
vethX-Y - virtual devices for host.
Packet sent to veth1-2 goes to chain and appears on veth3-2
Steps:
Recompile with:
CONFIG_RTE_LIBRTE_PMD_PCAP=y
ip link add veth1-2 type veth peer name veth2-1
ip link add veth2-3 type veth peer name veth3-2
ifconfig veth1-2 up -arp
ifconfig veth2-1 up -arp
ifconfig veth2-3 up -arp
ifconfig veth3-2 up -arp
mount -t hugetlbfs none /mnt/huge
Finally give l2fwd fake devices:
./l2fwd -c '0xf' --vdev "eth_pcap0,iface=veth2-1" --vdev="eth_pcap1,iface=veth2-3" -- -p 3
7. Upgrading ODP-DPDK to newer ODP API level
=================================================
This repository is based on odp.git, it also retains the history of that. There
are some modifications in configure.ac plus a few other files, but most of the
changes are in platform/linux-dpdk. That directory's Makefile.am builds our
code and the required parts from platform/linux-generic.
This allows us to easily pull the necessary changes from odp.git with git:
git remote add odp_base https://github.com/OpenDataPlane/odp.git
git pull odp_base master
This will result in a merge commit, and possibly some conflict resolving if
one of the files we touch outside platform/linux-dpdk is modified. After
resolving this conflict you need to implement the changes in the API. Also,
some of the code in our files are exact copy of the linux-generic counterpart,
it's important to port the fixes to the original. The git-transplant script
can do this:
scripts/git-transplant.py platform/linux-generic/ platform/linux-dpdk/ \
[the previous last commit merged from odp.git]..HEAD
It prints the list of prospective patches to be ported. See its comments about
what it does.
8. Building odp-dpdk with DPDK crypto PMDs
======================================================
Refer to the DPDK crypto documentation for detailed crypto PMD build instructions:
https://dpdk.org/doc/guides/cryptodevs/index.html
To build odp-dpdk with DPDK virtual crypto devices, we need to build supporting
Intel Multi-Buffer Crypto for IPsec library prior to DPDK build.
Get the Intel Multi-Buffer Crypto library from
https://github.com/intel/intel-ipsec-mb and follow the README from the repo on
how to build the library.
Building DPDK
-------------
Follow the instructions from "Compile DPDK" section. If libIPSec_MB has been
installed outside the normal search paths, configure the compiler and linker
options with:
meson configure -Dc_args=-I/path-to/Intel-multi-buffer-crypto/include \
-Dc_link_args=-L/path-to/Intel-multi-buffer-crypto/lib
Runtime parameters
------------------
When running an ODP application, include the required crypto devices in
ODP_PLATFORM_PARAMS environment variable.
E.g. ODP_PLATFORM_PARAMS="--vdev crypto_aesni_mb --vdev crypto_null"
9. Using eventdev scheduling and queues (experimental)
======================================================
ODP-DPDK includes experimental implementations of ODP scheduler, queues, and
scheduled pktio using DPDK event device library. The initial implementation
has been validated using only the standard software eventdev poll mode driver
with DPDK v18.11. Due to some pending eventdev bugs the implementation is not
yet tested in the ODP CI.
To use eventdev one must set ODP_SCHEDULER environment variable to "eventdev"
and provide the necessary platform parameters to DPDK.
Refer to DPDK event device driver documentation for platform details:
https://doc.dpdk.org/guides/eventdevs/index.html
In case of the standard software eventdev implementation one must enable a DPDK
service core, which will perform scheduling and receive packets for the
scheduled pktio input queues. The DPDK service cores and the ODP application
cores should not overlap.
Example how to run odp_scheduling test application using eventdev:
sudo ODP_SCHEDULER="eventdev" ODP_PLATFORM_PARAMS="--vdev event_sw0 -s 0x4" \
./odp_scheduling -c 1
10. Using dmadev for DMA transfers
==================================
ODP-DPDK implements ODP DMA API utilizing DPDK dmadev APIs. DPDK 21.11 or newer
is required, otherwise only a dummy implementation is provided. More
information about dmadev can be found under official documentation:
https://doc.dpdk.org/guides/prog_guide/dmadev.html
Before running applications, required DMA devices need to be set up. Official
documentation lists steps for setting up drivers for each supported device,
VFIO drivers typically being the least cumbersome to set up:
https://doc.dpdk.org/guides/dmadevs/index.html
As ODP DMA API provides a single capability for the underlying DMA machinery,
a set of similar capability devices is always tried to be discovered. If
application requires a certain device, this can be controlled with the normal
allowed/blocked device lists. E.g.:
sudo ODP_PLATFORM_PARAMS="-a 0000:f2:01.0" ./odp_dma_perf -t 1 -i 1 -o 1 -s 61440 -S 0 -m 0 -f 16
Additionally, a few configuration file parameters are available under the "dma"
section in DPDK configuration file. These should be configured according to used
device capabilities.