-
Notifications
You must be signed in to change notification settings - Fork 568
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
af_xdp sending with run to completion mode #398
Comments
There is no thread that does Tx in the SKB path of Linux. Only Rx is performed in a thread. If you are interested in zero-copy mode, then Rx and Tx will run in the same thread. The sendto() call is only there to make sure that this thread is woken up, but if you know you have Rx traffic, then there is no need doing this, though you will not send anything until you receive something. If you want the best possible performance but do not care about wasting cores, you can disable the need_wakeup mode. Without thiss mode, the Rx and Tx thread starts busy-spinning and goes on for ever. The only thing you need is either one sendto() call or receive one packet and it will start. Make sure the app and the interrupt (not that you will get any except for the first packet) are mapped to different cores. |
How can I disable need_wakeup mode? |
xsk_ring_prod__needs_wakeup is returning true. Is it something we can change? |
@Adarsh97 You can do this when creating the XSK (AF_XDP socket), example:
Are you seeing this last line (cfg.bind_flags) ? By default, in the example files it is defined like this: static u32 opt_xdp_bind_flags = XDP_USE_NEED_WAKEUP; Just remove it from your program, or better, if you are using the xdpsock example, The -m flag (according to the xdpsock example code) indicates that you should not use WakeUp, and unchecks the XDP_USE_NEED_WAKEUP flag |
So, to answer your question:
No, it is not by default, you need to define it explicitly, but in the example files it is already explicitly defined by default. |
Hi, `copy_mode_flg=0; // copy_mode int main() cfg.xdp_flags |= XDP_FLAGS_SKB_MODE; /* Set flag */ if (copy_mode_flg == 1) // zero_copy if(xdp_mode_flg == 1) // hw_offload bpf_obj = __load_bpf_and_xdp_attach(&cfg); xsk_configure_socket(umem, rx, tx,0,&cfg); } static struct xsk_socket_info *xsk_configure_socket(struct xsk_umem_info *umem,
}` kindly check above code and please let me know anything wrong i am doing here.
|
Here it worked normally, I didn't need to call kick_tx to send the packets. Please check with search (ctrl + f) if there is any part of the code that is setting |
My answers/questions: 1: If you are not using need_wakeup, you should ignore this flag. It is hardcoded to 1 if you are not using it, this so that applications that rely on it still work as expected. |
Hi, I am not specifically setting any flag ( i assume there is an automatic fallback present, if the highest one is not supported) so the sample code looks like this, while(1) xdp_send() without sendto calling tx_only function Observations are so temporary workround i found is call sendto in an interval, then it is working. But I know it is not optimal. The info of the system is as follows, |
You will get the default config parameters if you enter NULL as your cfg. If you specify any flags, then there is no fallback. What you specify is what you get. Please upgrade to a new kernel like 6.7 from kernel.org and try again. 4.18 plus whatever patches and backports RedHat puts on top of the kernel is really old and largely unknown in its content. Who knows if it works. You can verify if the kernel is constantly polling for packets without syscalls and interrupts by just doing a top and checking if the correct ksoftirqd threads is at 100% load. You could also use the xdpsock sample with the -m option and see if you get the desired behavior with that. |
@magnus-karlsson Please correct me if I'm wrong There are some exceptions, for example, in the Linux source code itself, there is an observation saying that if the driver does not implement the functionality, you must always use poll or sendto. Look:
I believe that virtio_net does not implement this functionality, according to PATCH (https://www.mail-archive.com/netdev@vger.kernel.org/msg406405.html) we can see that the final version has not yet been released. And also in this patch, we see that a zero copy version of XDP will be implemented for the virtio-net driver That said, @Adarsh97 is your network driver virtio_net? |
Hi, |
More specifically |
I can't tell if the i40e driver is up to date, have you tried updating the kernel to the latest version? |
If you would like to write a program that works in any mode, then the statement above is correct. You always have to call sendto() (or poll()) as the sending is not threaded in skb mode. If you, however, know you are in zero-copy mode, you can "cheat" and only call sendto() once as the softirq thread spins forever. Well, at least for real physical drivers.
|
Hi, |
Plz try these commands:
|
I have tried that. |
Plz try these commands:
Note "-dev" |
And when linking the .o files, don't forget to put -lbpf and -xdp |
@Adarsh97 It worked out ? |
How can we send (tx) packet using af_xdp with run to completion mode. All the example I am seeing are with a sendto option. (Which is not the one with best performance in a multi core system)
Do we have an example of tx having run to completion model.
The one system I am using is not sending packet, when the kick_tx (sendto call to kernel) is not called. I haven't set any need_wakeup flag. Still the ring return wakeup_required true.
How to resolve this?
The text was updated successfully, but these errors were encountered: