Elliott C. Back: In Aere Aedificare

I’ve just added two little cute features to the main page of my website. They don’t do much more than improve the usability and aesthetic of the front page by a tiny margin. The first is quite practical–it alerts you and sets the 404 status code if you loaded my site through a domain or subdomain:

The second is a Flickr badge across the top of my page, with a custom-made Flickr logo to take you to my Flick page:

But this is Web 2.0, and I use the Thickbox script in other places on my site, so why not here too?

It’s fun tinkering around with your main page. I need to add a cookie-rotator to the image on the front page, rather than make it time based. Then people can see different versions of me everytime they come back, rather than the current “new elliott at 1 AM” business.

I got a lovely email just now threatening me for being a notorious spammer:

Your doing it to drive up your Google Rank is pitiful, though I’ve informed Google of your attempts to game their system. Further evidence of scraping will be dealt with through the legal system. Perhaps a note to [your employer] will be of use as well.

I sent back my reply, which indicates that no I am not a spammer, thank you very much:

I’m terribly sorry you are experiencing web scrapers, but honest-to-god it’s not me. I wrote a plugin a long time ago for Wordpress called “WP Autoblog” that can take an RSS feed and import them as a series of posts. The posts get branded with attribution like “Post by XYZ and software by me” which you’re probably mistaking for something I’m actively a part of. I wrote the plugin to aggregate some of my family blogs (ericback.com, elliottback.com) together into a single feed, but it quickly became abused by spammers so I pulled it. You can read more here.

All this in spite of people making photo-aggregators, sitewide tagging, and making Planet sites. I can’t believe how much grief a hacky Wordpress plugin has given me over the years. Hopefully as it gets more and more out of date, this query count will start to drop from 400k (not that much) to a few hundred. Then I will smile.

Today I had the pleasure of a random guy in Mexico recursively downloading as much of my site as he could, which sent my CPU load to 2.0, a level that Dreamhost would find acceptable but which I personally freak out about. The r-dns and IP of this guy are:

dsl-189-171-15-59.prod-infinitum.com.mx
189.171.15.59

He started at 04/Nov/2007:12:04:36 and ended (by iptables ban) at 04/Nov/2007:20:17:03. In those 8 hours and thirteen minutes, he made over 250,000 requests. That’s an extra 8.5 requests per second from a single IP, which is clearly unacceptable behavior:

[root@fc624389 ~]# cat access_log | grep 189.171.15.59 | wc -l
251923

If you don’t believe me, the next biggest offender over the last 24 hours made only 4,400 requests:

[root@fc624389 ~]# cat access_log | cut -d’ ‘ -f1 | sort -n | uniq -c | sort -nr | more
251923 189.171.15.59
4403 66.249.73.116
2012 76.88.78.239
1646 70.141.105.233

The user agent of this guy doesn’t tell *me* anything about him, but maybe one of you readers has an idea?

189.171.15.59 - - [04/Nov/2007:12:04:38 -0500] “GET /wp-content/themes/greenmarinee/images/links_bullet.gif HTTP/1.1″ 200 467 “http://celebrity-photos.elliottback.com/” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Media Center PC 3.0; .NET CLR 1.0.3705; .NET CLR 1.1.4322)”

Another thing that bugs me is he requested each URL about 7 times. WTF? Do you really need to spider my site as fast as you can seven times?

[root@fc624389 ~]# cat access_log | grep 189.171.15.59 | cut -d’ ‘ -f11 | sort | uniq | wc -l
35414

I am either thinking of writing a very evil script to confuse non-google/msn/live/ask/yahoo bots by writing in an infinite number of invisible links into my websites, or installing some kind of mod_throttle into my apache. It looks like mod_limitipconn might help here, too.

If you are reading this, we just moved everything to a new server. The box is a Core 2 Duo 64bit machine w/ 2 GB of RAM, and a 160GB ATA hard disk, hosted by our buddies at Cari.net, the best web hosts ever!! If you’re wondering how we plan to transition, we moved all the files and DNS to this machine, and the DB is hosted still remotely on the old box. When the DNS has solidified, we’ll just scp over the database and everything will be 100% good to go.

There’s some weird downtime today on many of my sites, due to a DDOS on my provider:

Elliott, as of around 2:30PM PDT (GMT-7) one of Cari.net’s data centers was hit by a large scale DDoS attack. Our networking team has been working as quickly as possible to resolve this issue. The attack has been isolated and null routed, however, we are currently awaiting re-synchronization with our upstream providers. We are very sorry for any inconvenience this issue may have caused or is currently causing. Be assured, we are doing everything within our power to resolve this as quickly as possible.

I hope it gets fixed soon…

Update: I got a nice email explaining things:

On Saturday afternoon July 14th, 2007 a customer located within the Cari.net network was targeted by a massive DDoS (Distributed Denial of Service) attack. This attack reached magnitudes of 2Gbps at times and was carried out in a very sophisticated manor. This was one of the largest attacks seen in our company history.

The Cari.net networking team was onsite within minutes of the initial report. The first priority was to determine the target of the attack. Once this was done, the target was taken offline and security measures were implemented to try and stop the spread of the attack. After all steps had been taken locally to stem the attack, the Networking team contacted Cari.net’s upstream carriers and enlisted their assistance to “black-hole” the offending IP addresses from which the attack was believed to have been originated.

Even with these measures in place the attack still caused issues on the network. Cari.net engineers adjusted traffic routes and even removed entire upstream paths in an effort to stem the residual effects of the attack.

In the early morning hours of July 15th, the attack was considered under control but lingering effects could still be felt depending on where specific internet traffic was destined. Cari.net network engineers continue their work today to resolve all remaining issues and to return network performance to 100%. Minor BGP recalculations can be expected during the daytime hours Sunday as upstream connection and routes are brought back online.

For those that have experienced them, DDoS attacks can be one of the most difficult networking issues to deal with. Cari.net has been making modifications to our network to allow us the ability to deal with these issue more effectively for the past 60 days. These changes include:

1. Deploying our dual Proventia G2000 series IDS/IPS devices to cover a larger portion of our network. When complete, we will be able to provide more protection and enhanced defense against future DDoS attacks.
2. Only 10 days ago we purchased an additional OC48 of traffic from AT&T. This connection will be live within the next 60 days and will add even greater connectivity to our current BGP blend.
3. We are also upgrading other upstream connections to OC48 size so that we are able to handle any eventuality that may arise in the future.

With these changes in place, Cari.net’s network will be more robust than ever before.

We appreciate your patience as we worked through this issue. Should you still have any concerns or issues, please open a case with specific details and our on-site networking team will attend to your requests.

Thank you.

Chris Orlando - Vice President
Cari.net

This is why I use Cari.net and not Dreamhost–I get the impression they know what they’re doing.