We'r back ... again ...

Something happened early in the AM and our db was reset to May 3 .  But it's back now .

I'll let you know if there is anything about this epasode that is worth blogging.

Woops ... this happend again this morning around 5 Am, Jun 7th 2006.   Apparently what happened is somebody at Spry did some work on the computer which holds our old system before the migration to the current hardware node and then rebooted.  That changed the routing of our IP address to the old machine.   So the old hardware configuration was processing fastblogit.  When logged into the control panel you saw the correct data because the control panel is not using the same routing configuration as the web server and ftp server for this domain.  Wierd and confusing, i know, but no data was lost and we are back up and running now.

Tags

  1. system
  2. logs
  3. server crash

Comments


Seth says
M 2006-06-07 12:28:32 3667
Hopefully they won't do it again tomorrow.
Joseph said that he would fix it so that it won't.

Seth says
M 2006-06-08 08:27:45 3667
seth 2006-06-07 12:30:37 3667
M 2006-06-07 12:28:32 3667
Hopefully they won't do it again tomorrow.
Joseph said that he would fix it so that it won't.
killing the server completely wasn't what I had in mind
Ther server ran out of memory and stopped.  There were many unkilled pop processess running on it.  This did happen before.  I'm not quite sure what these unkilled pop process are.

Mark de LA says
I was just using your phrase "Telnet attack" as the cause. I don't really know what it is.

Seth says
M 2006-06-10 06:23:37 3667
I wonder what happened this morning around an hour ago ?
  • i had to restart the server. 
  • things stop when we run out of memory.  
  • around 5:30 we had zillions of legitimate pages accessed by a bot from ip 66.249.66.13.  These showed up on the http error logs. 

Mark de LA says
Looks like a googlebot - see ip trace below from uri http://visualroute.visualware.com/




Mark de LA says
I wonder if google is crashing our server 


Seth says
...
  • the bot reported it's self as "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
  • these were hitting us about one a second
I had though that legitimate bots were not suppose to hit a site that hard. 
source: google
For most sites, Googlebot shouldn't access your site more than once every few seconds on average. However, due to network delays, it's possible that the rate will appear to be slightly higher over short periods.
...
source: google

Googlebot is crawling my site too fast. What can I do?

Please contact us with the URL of your site and a detailed description of the problem. Please also include a portion of the weblog that shows Google accesses so we can track down the problem quickly.

...
...






Mark de LA says
M 2006-06-10 07:00:51 3667
I wonder if google is crashing our server 
Even more paranoid, noticing that it goes thru D.C. & Virginia (Langley ?) - I wonder if the NSA is monitoring it

Mark de LA says
Also on Google our site now lists only 35,000 items verses the quarter million or more before.

Seth says
M 2006-06-10 07:05:07 3667
M 2006-06-10 07:00:51 3667
I wonder if google is crashing our server 
Even more paranoid, noticing that it goes thru D.C. & Virginia (Langley ?) - I wonder if the NSA is monitoring it
I'm not sure it goes through Virginai ... it looks to me like the web service had to hop around all over the place just to match the ip to Google at Mountainview.

Seth says
... anyway, according to the system messages log, the memory overage occured at 4:17 and not around 5:30
source: logs/messages
Jun 10 04:17:24 robustai syslogd: select: Cannot allocate memory
Jun 10 04:17:55 robustai last message repeated 490928 times
Jun 10 04:18:14 robustai last message repeated 142573 times
Jun 10 04:19:07 robustai popa3d[8087]: fork: Cannot allocate memory
Jun 10 04:19:10 robustai popa3d[8107]: fork: Cannot allocate memory
Jun 10 04:19:10 robustai popa3d[8109]: fork: Cannot allocate memory
Jun 10 04:19:10 robustai popa3d[8111]: fork: Cannot allocate memory
Jun 10 04:19:10 robustai popa3d[8113]: fork: Cannot allocate memory

...



Seth says
Then here we were under a Telnet attack
source: logs/messages
Jun 10 04:48:20 robustai sshd(pam_unix)[22244]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=24.83.212.148  user=root
Jun 10 04:48:22 robustai sshd(pam_unix)[24534]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=24.83.212.148 user=root
Jun 10 04:48:22 robustai sshd(pam_unix)[25627]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=24.83.212.148 user=root
Jun 10 04:48:23 robustai sshd(pam_unix)[26296]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=24.83.212.148 user=root
... many more
... and this is me restarting the server:
source: logs/messages
Jun 10 05:28:29 robustai sshd(pam_unix)[26040]: session opened for user root by (uid=0)
Jun 10 05:30:14 robustai httpd: httpd shutdown succeeded
Jun 10 05:30:19 robustai httpd: [Sat Jun 10 05:30:19 2006] [warn] VirtualHost 66.249.1.177:443 overlaps with VirtualHost 66.249.1.177:443, the first has precedence, perhaps you need a NameVirtualHost directive
Jun 10 05:30:19 robustai httpd:
Jun 10 05:30:19 robustai httpd: [Sat Jun 10 05:30:19 2006] [warn] VirtualHost 66.249.1.177:443 overlaps with VirtualHost 66.249.1.177:443, the first has precedence, perhaps you need a NameVirtualHost directive
Jun 10 05:30:19 robustai httpd: [Sat Jun 10 05:30:19 2006] [warn] VirtualHost 209.35.87.155:443 overlaps with VirtualHost 209.35.87.155:443, the first has precedence, perhaps you need a NameVirtualHost directive
Jun 10 05:30:24 robustai httpd: httpd startup succeeded

...




Mark de LA says
an IP trace is the route the server takes to get to an IP - not a problem with the web service. It can be done using the command line independent of the web service. 


Seth says
And from the web server log around the time of the memory overage ...
source: /..../fastblogit.com
85.216.146.187 - - [10/Jun/2006:04:34:19 -0700] "GET /myfastblog/ HTTP/1.1" 200 4249 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
141.155.163.197 - - [10/Jun/2006:04:34:19 -0700] "GET /the%20great%20work/tags/projective%20geometry,gentle%20rain HTTP/1.1" 200 14054 "http://www.cfrussell.homestead.com/files/contents.htm" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/418 (KHTML, like Gecko) Safari/417.9.2"
220.73.146.126 - - [10/Jun/2006:04:34:19 -0700] "GET /tags/%24wish%2Cuser HTTP/1.1" 200 16076 "-" "NaverBot-1.0 (NHN Corp. / +82-31-784-1989 / nhnbot@naver.com)"
220.73.146.208 - - [10/Jun/2006:05:30:39 -0700] "GET /tags/signin/offset/6 HTTP/1.1" 200 5782 "-" "NaverBot-1.0 (NHN Corp. / +82-31-784-1989 / nhnbot@naver.com)"
24.22.135.240 - - [10/Jun/2006:05:30:40 -0700] "GET /river/ HTTP/1.1" 200 52945 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.4) Gecko/20060508 Firefox/1.5.0.4"

  • Apparently the NaverBot was the last thing to access us at 4:34
  • note the lack of activity between 4:34 and 5:30 when i brought the server back up
The Google bot hit us 8 minutes later.


Seth says
One thing i'd like to do is analyze the server logs and try to figure out why we jumped from 300-500/visits/day in mid May to 1000-16000/visits/day in late May till present.  This was a dramatic increase and there doesn't appear to be any reason for it.  


Seth says
M 2006-06-10 07:33:33 3667
an IP trace is the route the server takes to get to an IP - not a problem with the web service. It can be done using the command line independent of the web service. 
i bet if you do it from there, your trace rout will start out in Colorado. I'm just saying the web service started out in Washington where their web server is.

Seth says
M 2006-06-10 07:36:19 3667
I wonder why anyone would want to do a denial of service attack! fbi is not famous yet.

I don't see any evidance for a denial of service attack.  All i see is normal bot activity and somebody trying very hard to break into the server via Telnet.

See Also

  1. Thought Server Crash 2016-09-24 with 22 viewings related by tag "server crash".
  2. Thought Saturday Breakfast with 21 viewings related by tag "ServerCrash".
  3. Thought In Honor of the Server with 9 viewings related by tag "server crash".
  4. Thought server crash with 1 viewings related by tag "server crash".
  5. Thought Server Crash with 0 viewings related by tag "server crash".
  6. Thought server crash with 0 viewings related by tag "server crash".
  7. Thought Hippie Pioneer Lumber with 0 viewings related by tag "logs".
  8. Thought server crash with 0 viewings related by tag "server crash".
  9. Thought server crash with 0 viewings related by tag "server crash".
  10. Thought Sever crash with 0 viewings related by tag "server crash".
  11. Thought The server crashed with 0 viewings related by tag "server crash".
  12. Thought Server crash with 0 viewings related by tag "server crash".
  13. Thought server crash with 0 viewings related by tag "server crash".
  14. Thought ssh attack with 0 viewings related by tag "server crash".
  15. Thought server crash with 0 viewings related by tag "server crash".
  16. Thought growing pains with 0 viewings related by tag "system".
  17. Thought server crash with 0 viewings related by tag "server crash".
  18. Thought server crash with 0 viewings related by tag "server crash".
  19. Thought Hosting Support with 0 viewings related by tag "server crash".
  20. Thought server crash with 0 viewings related by tag "server crash".