Server outages

From time to time something on the new server runs amok. Ping then is still working, ssh login and all other I could think of is not. Most of the times everything comes backup OK after some minutes, but I had to reboot the server twice. And the last outage took MySQL down.

The weather is fine, I am sitting on my balcony, drinking tea, eating cookies and reading “Zero History” by William Gibson. So I will will not try to work this out just now. ;-)

Update (Oct 3 18:12):

I found the reason after thinking a lot about things that occur every 6 hours on my machine. Nothing – but then I filtered the access log of the server. A massive parallel scraping of web pages from one IP. I have blocked that IP now and am thinking about limiting the number of parallel connections of one IP.

Update again: Complained at amazon web services.

Update Oct 5: Reply from Amazon:

Hello,

This case has been investigated and resolved by the Amazon EC2 Abuse Team. Instance has been terminated.

Thank you for your attention in this matter.

Regards,
Amazon EC2 Abuse Team

 

Terminated. This somehow gives me a mental image of a guy in trench coat and his face in the shadow of his hat leaving the scene….

 

One thought on “Server outages

  1. Hi Rolf :).

    Servers servers servers… I got myself into a bit of trouble with my server as well last weekend. My internet connection is weird (that’s the understatement of the century Dude…)

    A) I use http://three.ie as a internet connection provider
    B) I have 3G modem connected to my server via USB extension cable
    C) 3G Huawei E160G modem is being dialed to the internet thanks to PCC (PCLinuxOS Control Center – rebranded Mandriva Drak Tools)
    D) My server stands on the wardrobe and it provides:

    a) First and main firewall for my network (passed http://grc.com stealth shields up test)
    b) Internet Connection Sharing server
    c) DNS server for my LAN computers
    d) Printer / scanner server for my LAN computers
    e) Apache HTTP server with WordPress blog on it (slow but working)
    f) Remotely controlled SSH server from WAN and LAN side
    g) LAN IRC server
    h) LAN IRC Services
    i) Pop3s and smtps e-mail server for LAN computers
    j) Fail2ban brute-force preventing barrier
    k) No-ip client for the http://no-ip.com – provides me with no-ip domain for my site.
    l) Connection keep alive script
    m) IRC bot for my irc channel

    E) My server has 2 LAN cards. One of them is used to share the 3G internet from my server to my Linksys router.
    F) Router spreads the internet via wireless link to my 4 laptops and other devices.

    Lately my server goofed off completely. I had internet on the sever but depending on the server’s sense of humor I had no internet / only google / all the sites but google on my client machines… I thought it was my fault… After 3 days of messing with it I got help from a very nice fella on IRC. I had to change the line:

    CLAMPMSS=NO

    from NO to YES in the /etc/shorewall/shorewall.conf file as my ISP decided to break the ICMP packets fragmentation… 3 days – Friday, Saturday, Sunday wasted to find the cause of this… BUT finally I was back online…

    Just thought I would share this as a server related anecdote / story / maybe solution for someone else ;).

    Cheers.

    Andy

Anything to add from your side of the computer?