Node server shuts down and cannot restart

[wchen-r7]

So for some reason, the node server sometimes can shutdown and not restart, and the browser gets no more testcases to fuzz. However, the debugger and continue.exe are still alive.

I don't really know why this is happening, but as far as I can tell, I can only guess it's possibly due to one or more of these contributing factors: 1) Low resources, 2) Unsafe threading, 3) Some sort of bad timing? Once again, I'm just guessing so I may be completely wrong the contributing factors.

I actually setup two nodes to test the first theory - one has 1GB of RAM, the other has 2GB. The 1GB RAM setup eventually hits the problem, but the 2GB setup is still alive. So I kind of feel maybe this issue has something to do with low resources.

I also know I'm actually not alone with this issue. Our friend @pyoor has seen something similar, so I would like to invite him to this discussion and maybe better document the problem (or not)

[wchen-r7]

Haven't really figured out why this happens, but a solution that works is to reboot your box periodically with Task Scheduler, and then automatically run Grinder.

[mutfuzz]

actually i have faced the same problem myself, and i noticed that it
happens when the vm has low resources, what i did at that time is to test
the server (ie: send a request) and wait for response, if i do not get a
response, then i will restart the server

i remember that i have added this patch in grinder.rb, but now it seems
that i have lost it since last update, however it should be trivial to
implement

On 12 June 2014 21:23, sinn3r notifications@github.com wrote:

Haven't really figured out why this happens, but a solution that works is
to reboot your box periodically with Task Scheduler, and then automatically
run Grinder.

—
Reply to this email directly or view it on GitHub
#40 (comment).

[stephenfewer]

I see this too on all of my systems which I have moved over to ruby 2.0.0. I don't recall this issue manifesting on ruby 1.9.3 (AFAIK). Will investigate more :)

[wchen-r7]

That'd be great, thanks!

[wchen-r7]

I have submitted a patch to make sure grinder can restart the node server if it's down for some reason. If you approve that patch, please still keep this issue open because that patch does not address why the server goes down sometimes. (And yes, I am still having that problem). Thanks.

[wchen-r7]

@stephenfewer,

A while ago I spent some time trying to debug this problem. And I noticed when the server was being killed, occasionally I'd get a backtrace due to a nil in @thread in server.rb. I can't find the backtrace at the moment, but I wanna say it was @thread.shutdown that hit it. I'm guessing this might be why it causes the process to terminate completely, but grinder doesn't know this happening so it keeps the invalid PID. Because of this condition, this if will never trigger:
https://github.com/stephenfewer/grinder/blob/master/node/grinder.rb#L182

And if that block doesn't trigger, you can't reset the server. No server = no testcases.

#46 will help tracking this state, and recover from it. But if my root cause analysis is correct, I guess a better fix would be improving managing the thread?