Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CPU usage still at 99% on last revision #15

Open
GoogleCodeExporter opened this issue Mar 15, 2015 · 10 comments
Open

CPU usage still at 99% on last revision #15

GoogleCodeExporter opened this issue Mar 15, 2015 · 10 comments
Labels
auto-migrated Priority-Medium Type-Defect

Comments

@GoogleCodeExporter
Copy link 

Hi

tested today the last revision but it seems the problem is not solved :

#slowhttptest -c 6000 -X -r 200 -w 512 -y 1024 -n 5 -z 32 -k 8 -u 
http://testurl -p 10 -l 6000

- during initialization phase the cpu stays at around 35% and pending 
connections ramp up :

Fri Sep 21 13:26:37 2012:slow HTTP test status on 30th second:
initializing:        0
pending:             3527
connected:           28
error:               0
closed:              0
service available:   NO
Fri Sep 21 13:26:42 2012:slow HTTP test status on 35th second:
initializing:        0
pending:             4133
connected:           28
error:               0
closed:              0
service available:   NO
Fri Sep 21 13:26:47 2012:slow HTTP test status on 40th second:
initializing:        0
pending:             4781
connected:           28
error:               0
closed:              0
service available:   NO
Fri Sep 21 13:26:52 2012:slow HTTP test status on 45th second:
initializing:        0
pending:             5412
connected:           28
error:               0
closed:              0
service available:   NO
Fri Sep 21 13:26:57 2012:slow HTTP test status on 50th second:
initializing:        0
pending:             5972
connected:           28
error:               0
closed:              0
service available:   NO


- but after all 6000 connections are in pending mode, the cpu usage is now 99% 
and closed connections start increasing:

Fri Sep 21 13:27:12 2012:slow HTTP test status on 65th second:
initializing:        0
pending:             5924
connected:           28
error:               0
closed:              48
service available:   NO
Fri Sep 21 13:27:17 2012:slow HTTP test status on 70th second:
initializing:        0
pending:             5364
connected:           28
error:               0
closed:              608
service available:   NO
Fri Sep 21 13:27:22 2012:slow HTTP test status on 75th second:
initializing:        0
pending:             4769
connected:           28
error:               0
closed:              1203
service available:   NO
Fri Sep 21 13:27:27 2012:slow HTTP test status on 80th second:
initializing:        0
pending:             4193
connected:           28
error:               0
closed:              1779
service available:   NO
Fri Sep 21 13:27:32 2012:slow HTTP test status on 85th second:
initializing:        0
pending:             3606
connected:           28
error:               0
closed:              2366
service available:   NO
Fri Sep 21 13:27:37 2012:slow HTTP test status on 90th second:
initializing:        0
pending:             2997
connected:           28
error:               0
closed:              2975
service available:   NO



- finally i reach a state where almost all connections whent from pending to 
closed, just 129 left and it stays that way for a long time with cpu at 99% and 
target DoSed:

Fri Sep 21 13:33:17 2012:slow HTTP test status on 430th second:
initializing:        0
pending:             129
connected:           28
error:               0
closed:              5843
service available:   NO
Fri Sep 21 13:33:22 2012:slow HTTP test status on 435th second:
initializing:        0
pending:             129
connected:           28
error:               0
closed:              5843
service available:   NO
Fri Sep 21 13:33:27 2012:slow HTTP test status on 440th second:
initializing:        0
pending:             129
connected:           28
error:               0
closed:              5843
service available:   NO
Fri Sep 21 13:33:32 2012:slow HTTP test status on 445th second:
initializing:        0
pending:             129
connected:           28
error:               0
closed:              5843
service available:   NO
Fri Sep 21 13:33:37 2012:slow HTTP test status on 450th second:
initializing:        0
pending:             129
connected:           28
error:               0
closed:              5843
service available:   NO
Fri Sep 21 13:33:42 2012:slow HTTP test status on 455th second:
initializing:        0
pending:             129
connected:           28
error:               0
closed:              5843
service available:   NO
Fri Sep 21 13:33:47 2012:slow HTTP test status on 460th second:
initializing:        0
pending:             129
connected:           28
error:               0
closed:              5843
service available:   NO
Fri Sep 21 13:33:52 2012:slow HTTP test status on 465th second:
initializing:        0
pending:             129
connected:           28
error:               0
closed:              5843
service available:   NO
Fri Sep 21 13:33:57 2012:slow HTTP test status on 470th second:
initializing:        0
pending:             129
connected:           28
error:               0
closed:              5843
service available:   NO
Fri Sep 21 13:34:02 2012:slow HTTP test status on 475th second:
initializing:        0
pending:             129
connected:           28
error:               0
closed:              5843
service available:   NO
Fri Sep 21 13:34:07 2012:slow HTTP test status on 480th second:
initializing:        0
pending:             129
connected:           28
error:               0
closed:              5843
service available:   NO
Fri Sep 21 13:34:12 2012:slow HTTP test status on 485th second:
initializing:        0
pending:             129
connected:           28
error:               0
closed:              5843
service available:   NO
Fri Sep 21 13:34:17 2012:slow HTTP test status on 490th second:
initializing:        0
pending:             129
connected:           28
error:               0
closed:              5843
service available:   NO

Original issue reported on code.google.com by sheepdes...@gmail.com on 21 Sep 2012 at 11:43
@GoogleCodeExporter
Copy link
Author 

Hi, thanks for the bringing this up again. On OSX, I don't see this behavior. 
At least with local apache server on the same machine, max CPU usage is 60% for 
me at peak time, when poll has to monitor max file descriptor count. 
Can you please share some more details on your use case? OS, 
target(local/remote), SSL/plaintext?

Original comment by shek...@gmail.com on 22 Sep 2012 at 6:12

@GoogleCodeExporter
Copy link
Author 

Hi,

I run a Fedora 17 64b,
I tested only remote servers,
one is a Synology nas webmanager on https, the other one is an IIS 7 running on 
windows 2008.
symptoms are in both case the same : no hight cpu use during ramping up phase 
but as soon as all desired sockets are at least in pending state, cpu use 
reaches the top.

Original comment by sheepdes...@gmail.com on 24 Sep 2012 at 7:43

@GoogleCodeExporter
Copy link
Author 

Hi, hopefully I fixed it now. At least on my fedora. Then found there is 
similar problem on mac, that still exist after fixing the fedora. There were 
two issues - poll() differences between osx and fedora, and improper handling 
of probe connection when system can't create any more sockets. 

And thanks a lot for your feedback! If you have time, can you please make sure 
it works for you, before I close the bug? Checkout trunk for the changes.

Original comment by shek...@gmail.com on 11 Oct 2012 at 4:35

@GoogleCodeExporter
Copy link
Author 

Hi, i tested last trunk, I tried on my Fedora 64b  and asking for 2000 
connections to a remote Synology NAS web interface but got the same results :
CPU use is low ( 5~15%) until all requested connections are at least in 
"pending" state; 
From there, when all the 2000 (or 3000) requested connections are in pending, 
connected or closed, CPU use reaches 99%.

With a backtrack 64b virtual machine I got a bit different result:
still asking for 2000 connections to the same remote system,
CPU use stays low even after all 2000 connections are in pending state and 
connected ones rise, but as soon as some of them goes from connected to closed, 
CPU use reaches 99% 


By the way,"initializing" state always shows 0, no matter what. Normal?

Original comment by sheepdes...@gmail.com on 13 Oct 2012 at 1:36

@GoogleCodeExporter
Copy link
Author 

it seems that CPU rises to the roof when both all connections are a least in 
pending and at least one is closed, on the Fedora system and on The Backtrack 
one.
i played with -r (from 50 to 200)  and -c (2000 - 3000) to change the timing to 
reach the "all at least in pending" state. 

Is there a test protocol i could follow to make sure of what is the cause / 
when it happens?

Original comment by sheepdes...@gmail.com on 13 Oct 2012 at 2:10

@GoogleCodeExporter
Copy link
Author 

Thanks for the response. It's frustrating already:) 
I'll add some debug messages and a argument to enable them. On my Fedora and 
OSX, problems start when OS can't allocate any more file descriptors. After my 
fix I don't even see any problems with 6K connections.
"initializing" used to indicate sockets in SSL negotiation state, but after 
some changes in the logic, it's pretty much obsolete state and is always 0. 
Thanks for noticing, I'll remove it. 
P.S. Can you run same thing with some small number of connections? Say 1000.

Original comment by shek...@gmail.com on 14 Oct 2012 at 2:11

@GoogleCodeExporter
Copy link
Author 

with 1000 connections, it's hard to see when closed connections play a role or 
not as it can be quite fast,

anyway, it seems that when targeting IIS7, CPU overuse happens as soon as all 
1000 connections are at least in pending (so some are already connected) even 
with no closed ones

and when targeting Synology NAS, it seems that cpu use only reaches 99% from 
the first closed connection (but i may be wrong here and the trigger could be 
the same as the first example).

Same results on Fedora and Backtrack

Original comment by sheepdes...@gmail.com on 14 Oct 2012 at 5:43

@GoogleCodeExporter
Copy link
Author 

[deleted comment]

@GoogleCodeExporter
Copy link
Author 

i tested with a remote apache server, with Fedora or the Backtrack virtual 
machine.
As soon as all 1000 connections are pending or connected, CPU use goes up. That 
is before any closed connections.

Original comment by sheepdes...@gmail.com on 14 Oct 2012 at 6:43

@GoogleCodeExporter
Copy link
Author 

Hi, 
I upgraded my fedora system to version 18 with kernel 3.8
i did a new test against a nginx server with up to date slowhttptest and the 
following command :

slowhttptest -c 2000 -X -r 200 -w 512 -y 1024 -n 5 -z 32 -k 8 -u http://testurl 
-p 10 -l 240

Full Cpu use is still a concern after a short while, any hope i can help to 
debug?

Original comment by sheepdes...@gmail.com on 12 Feb 2013 at 12:46

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
auto-migrated Priority-Medium Type-Defect
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@GoogleCodeExporter