Elliott C. Back: Technology FTW!

Downtime today

Posted in My Blog by Elliott Back on July 14th, 2007.

There’s some weird downtime today on many of my sites, due to a DDOS on my provider:

Elliott, as of around 2:30PM PDT (GMT-7) one of Cari.net’s data centers was hit by a large scale DDoS attack. Our networking team has been working as quickly as possible to resolve this issue. The attack has been isolated and null routed, however, we are currently awaiting re-synchronization with our upstream providers. We are very sorry for any inconvenience this issue may have caused or is currently causing. Be assured, we are doing everything within our power to resolve this as quickly as possible.

I hope it gets fixed soon…

Update: I got a nice email explaining things:

On Saturday afternoon July 14th, 2007 a customer located within the Cari.net network was targeted by a massive DDoS (Distributed Denial of Service) attack. This attack reached magnitudes of 2Gbps at times and was carried out in a very sophisticated manor. This was one of the largest attacks seen in our company history.

The Cari.net networking team was onsite within minutes of the initial report. The first priority was to determine the target of the attack. Once this was done, the target was taken offline and security measures were implemented to try and stop the spread of the attack. After all steps had been taken locally to stem the attack, the Networking team contacted Cari.net’s upstream carriers and enlisted their assistance to “black-hole” the offending IP addresses from which the attack was believed to have been originated.

Even with these measures in place the attack still caused issues on the network. Cari.net engineers adjusted traffic routes and even removed entire upstream paths in an effort to stem the residual effects of the attack.

In the early morning hours of July 15th, the attack was considered under control but lingering effects could still be felt depending on where specific internet traffic was destined. Cari.net network engineers continue their work today to resolve all remaining issues and to return network performance to 100%. Minor BGP recalculations can be expected during the daytime hours Sunday as upstream connection and routes are brought back online.

For those that have experienced them, DDoS attacks can be one of the most difficult networking issues to deal with. Cari.net has been making modifications to our network to allow us the ability to deal with these issue more effectively for the past 60 days. These changes include:

1. Deploying our dual Proventia G2000 series IDS/IPS devices to cover a larger portion of our network. When complete, we will be able to provide more protection and enhanced defense against future DDoS attacks.
2. Only 10 days ago we purchased an additional OC48 of traffic from AT&T. This connection will be live within the next 60 days and will add even greater connectivity to our current BGP blend.
3. We are also upgrading other upstream connections to OC48 size so that we are able to handle any eventuality that may arise in the future.

With these changes in place, Cari.net’s network will be more robust than ever before.

We appreciate your patience as we worked through this issue. Should you still have any concerns or issues, please open a case with specific details and our on-site networking team will attend to your requests.

Thank you.

Chris Orlando - Vice President
Cari.net

This is why I use Cari.net and not Dreamhost–I get the impression they know what they’re doing.

Viewing 4 Comments

 

Trackbacks

(Trackback URL)

close Reblog this comment
Powered by Disqus · Learn more
blog comments powered by Disqus