Download limit problem

[davidkek]

Hi all

i have problem with set speed over 50mb/s

iptables -A FORWARD -m ratelimit --ratelimit-set up --ratelimit-mode src -j DROP
iptables -A FORWARD -m ratelimit --ratelimit-set down --ratelimit-mode dst -j DROP
echo / > /proc/net/ipt_ratelimit/up
echo / > /proc/net/ipt_ratelimit/down
echo +192.168.0.2 100000000 > /proc/net/ipt_ratelimit/up
echo +192.168.0.2 100000000 > /proc/net/ipt_ratelimit/down

192.168.0.2 cir 104857600 cbs 19660800 ebs 39321600; tc 0 te 0 last never; conf 0/0 0 bps, rej 0/0

problem is come when i try to set 70,80,90,100 mbit speed is limit to 8-10 mbit
after back shaper to 50 mbit is work fine shape customer to 50-52mb/s

how to debug this problem and to fix.

[aabc]

Hello. What is HZ value for your kernel? Try command zcat /proc/config.gz | grep CONFIG_HZ.

[aabc]

Please, better do zcat /proc/config.gz | grep _HZ, I want to see NO_HZ values too.

[davidkek]

Hi

CONFIG_NO_HZ_COMMON=y

CONFIG_NO_HZ_IDLE is not set

CONFIG_NO_HZ_FULL=y

CONFIG_NO_HZ is not set

CONFIG_RCU_FAST_NO_HZ=y

CONFIG_HZ_PERIODIC is not set

CONFIG_HZ_100=y

CONFIG_HZ_250 is not set

CONFIG_HZ_300 is not set

CONFIG_HZ_1000 is not set

CONFIG_HZ=100

this is a set of HZ config
and server work very fine
do you need to change from 100hz

[davidkek]

I try to reproduce in second lab system but not success
and wait to try with other device to see is have the problem or not .

[aabc]

Thanks. You may try to set higher HZ value, like 250 (reported to work good) or even more. Speed limiting arithmetic is 32-bit, so it could overflow with small HZ.

[davidkek]

did you think to remake it for 64bit.
however, it is more modern and more functional.
I know that I will have to redo not a small part of the code, but if I have to I will help.

[davidkek]

however, all new devices are 64bit and will be much more functional.

[aabc]

Also, CONFIG_NO_HZ_FULL=y I am not sure atm how that would affect speed calculation. It reported to work good on CONFIG_NO_HZ_IDLE=y though.

[davidkek]

i will try move to 250Hz and will test CONFIG_NO_HZ_IDLE=y

but in last test i prefer 100Hz and CONFIG_NO_HZ_FULL system work very stable and with low load .

[aabc]

did you think to remake it for 64bit.

Yes, probably, I should.

[davidkek]

i will wait this changes! they will certainly work much better.

[aabc]

i will try move to 250Hz and will test CONFIG_NO_HZ_IDLE=y

I would appreciate such test!

[davidkek]

Hi i am back
now i compile kernel on 250Hz and CONFIG_NO_HZ_IDLE=y

but the problem is hear and i find solution :)
when customer is set to 70Mbit or more now i test with 100mbit limit .
if machine is in idle and have 1 customer shaper and try speed test or other download from internet customer go to max 20mb/s from 15-20Mb/s
after run load software to load 2-3 cpu or more of the server shaper start work

Upload limit is fine
Only download limit have a problem .
without cpu load on server customer get 1 or 2 time 100mbit and after that get only from 8-20mb/s
after run cpu load software 3-4 cpus shaper start work fine 100 of 100mbit get on customer machine.

Very strange... machine run kernel 5.8.14 kernel and very clean image.
This is cpu : Intel(R) Xeon(R) CPU E5-1650 v4 @ 3.60GHz
Ethernet :
02:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
02:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

[aabc]

I think you should test locally (in local network), for example with iperf between two servers - not relying on customer reports or third-party internet speed testing services. If local test is correct, (and you say upload is ok), then ipt-ratelimit is working correctly and you, probably, have some other problems. If you measure between distant endpoints in Internet a lot of factors could affect the speed.

[davidkek]

i make the same test without ipt-ratelimit and with mikrotik device and work fine all test.

in case when run ipt-ratelimit and machine is 100% idle have problem when machine is loaded on 4 core or more shaper work very fine and this is strange
and only download limite have this problem is may be a kernel config problem or is a bug in ipt-limit ?

[aabc]

To localize problem we need to test in simpler to analyze and with less other factors, so please test in local network using iperf. Double check that you have set rules correctly.

[davidkek]

oki i find where the dog is buried

problem is come when i set uplink card to use rps and use all cpu .
card to custemer is set on all affinity and ok
when uplink card from inter net is set affinity and rps have problem.
When i set only IRQ to cpu is ok.
this is one big mystery ...

[Nable80]

Did I understand correctly that you're trying to shape incoming traffic? This is almost impossible in all cases (packets are already here, you can only drop or process them immediately), it may also make things even worse than lack of shaping: sender may overwhelm your card with re-transmitted packets, you'll see an IRQ storm and poor bandwidth usage. If you see high CPU load due to IRQ processing (IRQs are rarely used for TX operations but they're almost always used for RX queue), you may try to improve this situation by configuring IRQ coalescing. Unfortunately it's not trivial to find the most optimal settings (it's too individual), that's why I can't provide you exact options. You can try googling your NIC model with rx-usecs keyword.

[aabc]

This module is not shaper, but policer! It's possible to police at any point of the network (no matter where you drop packets - at the source, end, or the middle of the network path) - including incoming traffic, and it's used for exact such use case at many places with good reports.

[Nable80]

OK, sorry for this misconception. I still have some doubts that it's possible to avoid negative effects caused by senders which are trying to handle lost (not acknowledged) packets.

I'm also curious: how many pps (packets per second) were you able to process with this module on some general purpose hardware?

[aabc]

The module uses Cisco's RED-like TBF algorithm, which somewhat randomizes drops, this is exactly against correlated packet re-transmits (I think, it's possible to find Cisco or scientific papers about its performance). It should be possible to handle 10-Gbit (reported by many users years ago) on commodity server.

[aabc]

I don't remember exact pps values. I think you can divide bps by some average packet size. Because, ppl use not just in labs but in real ISPs too.