Server Loadavg high from two days ago

Obtain support and get your questions answered here.

Server Loadavg high from two days ago

Postby jimmybond » Thu Feb 11, 2016 6:32 am

According to my load average log file,it seems from two days ago liway's tesla server has experienced a prolonged high load,till now loadavg in 15min is greater than 50 which made the server much slower. Please see to this issue, is there some form of DDoS attack or other security problem? Thank you very much for keeping this free service run stable, I'd like to support liway by donation or some paid service(hopefully not too expensive :) )

Best,
Jimmy
jimmybond
New around here
 
Posts: 49
Joined: Sat Dec 11, 2010 7:55 am

Re: Server Loadavg high from two days ago

Postby jimmybond » Fri Feb 12, 2016 6:22 am

And another problem caused by server load is PHP mail function in Wordpress failed to work, not very sure why does this occur maybe some timeout settings of PHP mail?
jimmybond
New around here
 
Posts: 49
Joined: Sat Dec 11, 2010 7:55 am

Re: Server Loadavg high from two days ago

Postby Max » Fri Feb 12, 2016 10:41 am

I observed some high load as well a few days ago, though couldn't pin point what was causing it, beyond apache in general. I will have another look, though I haven't had that issue come up again in the last day.
Max
Administrator
 
Posts: 1031
Joined: Sat Jul 18, 2009 3:17 pm

Re: Server Loadavg high from two days ago

Postby jimmybond » Fri Feb 12, 2016 1:17 pm

Max wrote:I observed some high load as well a few days ago, though couldn't pin point what was causing it, beyond apache in general. I will have another look, though I haven't had that issue come up again in the last day.



Thank you for ur help! The server load has been ~10 around for a few hours till now. But it's still much higher than normal. From the records in my log files,the load average is about 3 for most of time,maybe it's a short term DoS attack if there is no other security issue? I'll keep watching the loadavg data then.
jimmybond
New around here
 
Posts: 49
Joined: Sat Dec 11, 2010 7:55 am

Re: Server Loadavg high from two days ago

Postby jimmybond » Fri Feb 12, 2016 3:15 pm

8.79 10.52 12.63 13/266 at 12/Feb/2016 14:00:02 UTC
8.38 11.17 12.60 11/283 at 12/Feb/2016 14:15:02 UTC
15.73 16.79 14.04 17/288 at 12/Feb/2016 14:30:01 UTC
56.36 46.56 30.87 58/316 at 12/Feb/2016 14:45:02 UTC --> High Load!!
54.92 55.72 46.80 65/339 at 12/Feb/2016 15:00:03 UTC --> High Load!!

This is loadavg log just now. From 2:30pm(UTC time), apparently there is a load spike. Hopefully this can help to locate the problem.
jimmybond
New around here
 
Posts: 49
Joined: Sat Dec 11, 2010 7:55 am

Re: Server Loadavg high from two days ago

Postby Max » Sat Feb 13, 2016 7:13 am

After some investigation, I believe this was actually my fault (kind of).

Baidu was hitting my images repository a lot, and unfortunately I have an index listing script which allows incorrect URLs, and will simply display the base tree index, so for whatever reason, baido went around adding random otherwise valid nestings of directory names to all valid urls on my site, which turned out to be invalid (ie. when put together), but presumably because it was getting a 200 back, it just kept doing it, thinking they are somehow different pages. So tl;dr: Baidu was recursively spidering my site and ddosing the server as a result.

I've removing my index listing script for now (I'll re-write it later to prevent this issue), so the load should now be normal.
Max
Administrator
 
Posts: 1031
Joined: Sat Jul 18, 2009 3:17 pm

Re: Server Loadavg high from two days ago

Postby jimmybond » Sat Feb 13, 2016 8:39 am

Max wrote:After some investigation, I believe this was actually my fault (kind of).

Baidu was hitting my images repository a lot, and unfortunately I have an index listing script which allows incorrect URLs, and will simply display the base tree index, so for whatever reason, baido went around adding random otherwise valid nestings of directory names to all valid urls on my site, which turned out to be invalid (ie. when put together), but presumably because it was getting a 200 back, it just kept doing it, thinking they are somehow different pages. So tl;dr: Baidu was recursively spidering my site and ddosing the server as a result.

I've removing my index listing script for now (I'll re-write it later to prevent this issue), so the load should now be normal.


Thanks a lot! All server data come back to normal now, this is current status in Cpanel:
Server Load 0.14 (4 cpus)
Memory Used 19 %
Swap Used 22.18 %

I also had a online proxy script deployed on a cloud platform which was once indexed by Xunlei (a popular Chinese download tool), Xunlei spider tried to download files with the script at nearly 2~3 requests/s and this ddos pulled my small app offline from time to time. I had to change the app's http response from 200 to 404 for Xunlei agent to solve the issue.

Cheers,
Jimmy
jimmybond
New around here
 
Posts: 49
Joined: Sat Dec 11, 2010 7:55 am


Return to Support & Questions

Who is online

Users browsing this forum: No registered users and 1 guest

cron