Elliott C. Back: In Aere Aedificare

Safari has a new version, and a new platform. The once mac-only web browser has released Safari 3 for Windows XP, backed by claims of UI, performance, and integration superiority. The claims are that it is 2x faster in Javascript and HTML rendering, a claim we plan to test ourselves. Note that Safari, the first time it was launched, took about a minute to start up and froze the screen for that time. Only after that did it open fast.

Javascript Performance Results

So, their biggest claim is Javascript performance, which they show like so:

We’ll combine the overall results from these benchmarks together:

1. JS/Bench
2. DHTML Benchmark
3. DOM Query Test
4. JS Speed Tests

We find that depending how you look at it, Safari can actually be considered 3x slower than IE7, or roughly of equal speed. Here is an overall performance chart, with two columns–one is the raw average score, the other averages the worst-test group (three results) into one result and averages it:

Safari doesn’t break much ground here

Here’s the per-test chart, which shows Safari kicking ass in the first test, losing the next three, tying the fourth with IE, and doing well again on the last test:

Other annoyances

Interestingly, Apple tries to bundle Safari with not one, but three separate Apple products: Quicktime, Bonjour, and Apple Update. On top of that, they break the back-mousebutton click that I’ve become used to using in IE/FF, and use tons of my RAM main memory up. No one sums it up better than Dev Hints who notes that “Safari Isn’t the Beauty That Apple Likes To Claim.” It’s not bad, and it’s getting better, but there are still bugs to be worked out.

I just added mod_deflate to my server by using the following configuration:

LoadModule deflate_module modules/mod_deflate.so
<IFModule mod_deflate.c>
  SetOutputFilter DEFLATE
  SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary
  SetEnvIfNoCase Request_URI \.(?:exe|t?gz|zip|bz2|sit|rar)$ no-gzip dont-vary
  SetEnvIfNoCase Request_URI \.pdf$ no-gzip dont-vary
  BrowserMatch ^Mozilla/4 gzip-only-text/html
  BrowserMatch ^Mozilla/4\.0[678] no-gzip
  BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
</IFModule>

The static parts of my pages now render with about 70% compression, meaning that I’m saving 2/3 of static page bandwidth now. Server load seems fine!

I’ve concluded that Dreamhost sucks phenomenally at hosting websites that generate any kind of traffic. Sure, their $9.99 a month plans with massive savings coupons are enticing, but if you knew what you were getting yourself into, you’d stay away. Dreamhost sucks like you’d want to suck on a knife covered in chocolate–which isn’t very much.

Others tell the tale better than I can:

• Amazon S3 v.s. Dreamhost
• Why Dreamhost Sucks
• Dreamhost Hosting Sucks Big Time
• Dreamhost getting sucky PR out on blogs
• Boo Dreamhost
• Dreamhost is REALLY STARTING TO SUCK
• Trash Dreamhost here
• Nightmare Host (From Hell?)

There are also two unbelievable sites which actually claim that Dreamhost doesn’t suck. Well, if you read the above articles, you’d understand that Nightmarehost is really a bad dream.

So far not a single blog has explained at a high technical level why Dreamhost can’t handle their customers. I’ve seen some vague hand-waving about overselling, but no one actually has numbers to back it up. Sure, when someone tells me it takes them 10 minutes to go from SSH login prompt to terminal I believe them, but it’s not good enough. We’re making serious accusations about quality of service; we had better be able to back it up. We need hard data.

I have a shared hosting account on one of their machines, sepulveda.dreamhost.com [205.196.222.24]. The ping is fairly responsive, but not exceptional. They get their bandwidth directly from Level3, so it’s good bandwidth:

%ping -n 100 sepulveda.dreamhost.com
Minimum = 77ms, Maximum = 116ms, Average = 87ms

Unfortunately, there are 1200 users on my machine. I’ve seen industry guidelines that recommend far, far less than that, anywhere from 1/4 to 1/10th for shared hosting services:

[sepulveda]$ cat /etc/passwd | wc -l
1199

The machine itself appears to be a single dual-core opteron with 4GB of RAM, which isn’t hefty by any means. It should be dual dual-core and have 16GB of RAM to be at all useful. Besides, RAM is cheap–if they did put in more RAM maybe they could realistically handle 1-2k users per machine! Here’s the proc info:

/proc/meminfo:

        total:    used:    free:  shared: buffers:  cached:
Mem:  4172861440 3993358336 179503104        0 24576000 2158034944
Swap: 6465036288 326541312 6138494976
MemTotal:      4075060 kB
MemFree:        175296 kB
MemShared:           0 kB
Buffers:         24000 kB
Cached:        2033020 kB
SwapCached:      74436 kB
Active:         810260 kB
Inactive:      1321512 kB
HighTotal:     3211200 kB
HighFree:        34404 kB
LowTotal:       863860 kB
LowFree:        140892 kB
SwapTotal:     6313512 kB
SwapFree:      5994624 kB

/proc/cpuinfo:

processor	: 0
vendor_id	: AuthenticAMD
cpu family	: 15
model		: 35
model name	: Dual Core AMD Opteron(tm) Processor 175
stepping	: 2
cpu MHz		: 2194.592
cache size	: 1024 KB
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 1
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic
sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse
sse2 ht syscall mmxext lm 3dnowext 3dnow pni
bogomips	: 4377.80

processor	: 1
vendor_id	: AuthenticAMD
cpu family	: 15
model		: 35
model name	: Dual Core AMD Opteron(tm) Processor 175
stepping	: 2
cpu MHz		: 2194.592
cache size	: 1024 KB
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 1
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep
mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht
syscall mmxext lm 3dnowext 3dnow pni
bogomips	: 4377.80

It’s funny that a guy actually monitored his site with Pingdom for a week. He writes in a comment on the dreamhost status blog:

Sorry, guys, but your service is simply terrible.
Today, there were just 83.11% of uptime (3h23min offline until now!) - data obtained from Pingdom.com.
Since 4/9, there were just one day with 100% uptime.
In the sum of last ten days, I just had more than 9 hours of downtime (while my other servers had no more than 15 minutes).
Every day I see server problems in my server.
That is unacceptable!!!

Dreamhost and I have been having conversations now for a while about a site which gets 1-2k visitors, and hosts 51GB of transferred static content a day. I thought you might be interested in reading them. On 4/30/2007, I received this email from Brian S. about my site:

Connections to your domain ( static.imgfly.com ) crashed the shared apache service several times this morning. A connection limit has been placed on your site. Being on a shared server means you need to share the resources with other customers. Due to the heavy volume of traffic, other domains on the same service were not able to load. Once the traffic to your site has taperd off, we will gladly remove the connection limit. Please read the appropriate section of our Terms of Service and let us know if you have any questions.

dreamhost.com/tos.html

My only thought is was dismal woe. If they don’t know how to configure Apache with the right connection and threading settings so it won’t crash, all is lost. Also, some of my DNS settings (I forget which exactly) were mangled in their blocking process, so I sent them a snarky email:

Unfortunately, you did far more than place a connection limit. You also edited my dns settings without my consent, which is strictly against any kind of ethical hosting policy. I had a CNAME on feifei.us pointing * to the domain so that subdomains would work; now it has mysteriously disappeared.

My account is advertised at having ~2.9TB of bandwidth a month; I pointed 50GB/day of static hosted content at my dreamhost account, which is about half of what I’m allotted, and you screw up my domain settings? You can’t handle half of what you promise?

I want to know exactly what kind of connection limit you’ve placed on my site, at a high technical level. I want to know why you can’t deliver even half of the bandwidth allotted to my account. This isn’t a high-level of bandwidth, it’s just a constant level of about 50GB a day, and it’s not even dynamic content, just static files.

At that time I was frantically re-routing and re-setting up the site, because all the DNS settings were lost, even ones pointing offsite and not at Dreamhost servers. They replied quite fairly to the email, I have to give them credit:

The CNAME was not removed as part of the connection limit. I’m sorry that it may have disappeared, but this was not as a result of the limit put in
place and was instead likely a bug in the system. You’re free to put it back if you’d like, but let me know if you don’t notice the wildcard working and I’ll get it set up for you again. I’m sorry that you believe we removed it, but we did not. The domain you mentioned was not one that was touched.

Bandwidth and connections are two separate issues. While you can definitely use all of the bandwidth that we offer, the number of connections per second and concurrent connections that your site was receiving was causing the Apache service to crash. If we didn’t put the limit in place you wouldn’t be able to use any of the bandwidth we provide as your site would have continued to crash the service. This would prevent yours and other customer’s websites from appearing online.

Unfortunately you’ve since removed hosting for this so I can’t provide you any sort of information about the limit that was put in place. I’ve explained the bandwidth issue above. Please don’t hesitate to write back if you need help with anything else.

Of course, I don’t buy the bandwidth explanation. They would have to allow me enough connections to actually constantly use 11Mbs for me to use it up. With 1200 customers on their server, even with a gigabit ethernet card they couldn’t fulfill their contracts if everyone used all their bandwidth. Some overselling is, of course, necessary and acceptable. Dreamhost goes a little overboard. The next day I noticed the new arrangements of sites I set up wasn’t working well, so I sent them a quick checkup email:

There’s some serious issues with this site; it took 8 connection tries to even connect to the server.

The reply I got back was shocking, even to me (emphasis my own):

Your domain imgfly-static.feifei.us is again causing serious problems on the sepulveda webserver. The number of requests coming in is almost instantly crashing the apache instance. This is unacceptable, and since instead of working with us you simply moved your problematic site to another domain I’m going to ask you now to stop running whatever you’re currently hosting on imgfly-static.feifei.us immediately, and to not start anything like it on any user, domain, or server of ours, ever again.

I’ve disabled imgfly-static.feifei.us to preserve the stability of the server, please do not re-enable it.

I only moved my domains around because (a) I wanted them that way in the first place, and (b) the DNS disappeared at some point. I had thought their connection limit was account-wide, but apparently my fixing the DNS also ruined their limiting. Today, since even though they suck, I sent them a nice, long, clearly written, conciliatory email. I really do want to get my 3TB of bandwidth out of them, and if it takes some sucking up, so be it:

I feel like we’re misunderstanding each other–I’m not trying to subvert your sepulveda cluster, and I am trying to work with you. Last time there was an issue, either my CNAME or A record for the subdomain somehow got lost in the dns, so I added a wildcard CNAME to feifei.us. I was under the impression that connection limits were placed on the account to prevent it from affecting sepulveda, but whatever change I made must have invalidated that.

I’ve just arrived home from work and I’ve pointed the stream of traffic you can’t handle elsewhere until we can work out the configuration. I’ve enabled the subdomain again, but there won’t be anything running on it until I get the ok.

Let me explain the software solution I’m running. The domain ImgFly.com is hosted on a dedicated server I run offsite. Requests for actual image content are forwarded to imgfly-static.feifei.us, which serves them as static content if they have been cached by that server, otherwise makes an attempt to fetch the resource remotely from amazon S3 and cache it to disk. Approximately 100 photos are uploaded an hour, and 2.1 GB of data downloaded. This isn’t much. It’s mostly serving random static content to consumers.

When I SSH into sepulveda, I see some indications that the problems you’re seeing aren’t my fault:

[sepulveda]$ uptime
16:40:39 up 26 days, 2:48, 5 users, load average: 10.67, 10.77, 9.30

Sepulveda is a dual-dual-core opteron, but those load averages are still pretty high. When I SSH in I can barely get a single command to run. Clearly the server is overloaded to the point where all it can do is serve an extremely limited amount of information. I’m willing to work with you to best manage your server resources, but you need to let me know exactly what you can handle.

Here are some solutions that come to mind:

1) Move me (or just feifei.us) to a reasonably loaded server
2) Configure apache so it doesn’t crash, or, give me an .htaccess file with reasonable limiting I can use
3) Run feifei.us without mod_rewrite or the cache script, just on a completely static filesystem
4) Tell me exactly how many connections you are able to handle, and I will send you only that many

I’m sure you’ll have some good ideas as well. It’s disappointing to be offered 3TB of bandwidth a month which is an unmetered constant rate of 10 Mbs, but be unable to fully utilize it.

I’m sure this post will get lots of comments… if any of you know someone willing to host 2+ TB of bandwidth for < $100 a month, let me know. I’m almost to the point of paying for another dedicated server to manage this.

Update:

I guess telling Dreamhost that I’d turn off the site which was causing them problems and work with them to figure out a better way to host it was a bad idea, because I received this lovely email a few moments ago:

Hello,

I’ve disabled your account for failure to comply with my request. This
is a permanent account closure, I’ve refunded your last payment.

James

So my DNS and whatever miscellaneous files (I think my dad’s site!) are there are currently being held hostage. Considering they’re my property, unless I hear from Dreamhost in the next three hours, I will be calling my lawyers tomorrow. I want blood now.

Update 2:

At 4/04/2007 5:19 PM EST I received a lovely email from Dreamhost, explaining that they weren’t doing anything to address the fact that my domains and data are being held hostage:

I have gone ahead and forwarded this to James, he will get back to you as soon as he can. Please wait for his reply.

I have forwarded the to XX so that they can update your incident report with this info, They will get back to you with any information that they may have if it is necessary, if so please wait for their reply.

It is now 24 hours since my first of three requests in writing for them to release my data and domains to me. If you know someone at Dreamhost who’s friendly and sane enough to let me pick up my things and leave, you should let them know about this. Otherwise, this is going to be a whole lot more painful tomorrow.

Update 3:

Some kind soul submitted this story to Digg. Yay! Maybe if it gets enough attention Dreamhost will give me my domains and access back.

Update 4:

Hello Digg crowd. Maybe I wasn’t clear about things. This site isn’t running on a Dreamhost machine. Hellllllll no. It would be down right now if it was. My Dreamhost account, which contains only domains, my dad’s low volume blog, and maybe one other blog, is currently disabled. The rest of it’s on a dedicated server I run from Cari.net, a company I’ve never had any problems with.

Update 5:

Now they’re telling me to wait. This should take a tech all of 2 minutes to resolve, just enable the account, wait for me to tell you I’m done with it, and then close it permanently:

It means that we have passed your message to the Tech that is responsible for disabling your site. While we understand your urgency to get this issue resolved, you will need to wait until the Rep is able to follow-up with you regarding the issue.

Update 6:

The Co-founder of Advection .NET emailed me and offered to help out. Really nice guy. If you need serious big time content-distribution and bandwidth, you should check out Advection .NET Global Media Hosting Network.

Update 7:

Ah, the sweetness of resolution. It’s a very long day and a half later, but I’m now almost again in possession of my files, databases, and domains. For some reason, the Dreamhost abuse team decided to zip up my files themselves. They probably didn’t trust me to download my files and be off again:

I’m sorry about the delay in this responce, it was in part due to the time needed to prepare all of your data.

I’ve tarred up your files and placed them at abuse.dreamhost.com/misc/user-content/xxx/ the login is xxx and the password is that which you used to login to the Webpanel. Your data has been split into three files, and will need to be concatenated before you can unzip it. Also there are your databases, they were spread between three database servers, so are in three different files. I will be removing the files after one week.

Here are the authorization codes for the domains you have registered with us. Once you’ve initiated a transfer out you can contact us again and we’ll approve the transfer.

If they had added a line to their two line email about terminating my account saying “We will provide your domains and files for transfer in __some timeframe__, please wait for our email” they could have averted this post.

Final Update:

I’ve finally gotten an email from someone in the know, a level 2 support manager. Hurray for moving up. His email is very nice:

I’m terribly sorry about the recent events that have transpired, it looks like we disabled you a little overzealously. I apologize for that. We’ve re-activated your account, however, it’s my understanding that you’re moving your domains off of our servers, which is completely understandable. If you’d still like to do that, we’ll be more than happy to refund that payment James offered.

As for the domain, imgfly.com, I’ve moved it over to your account, so you should be able to transfer. To save you time, I’ve included your auth code for transfer, should you need it. Now, if you do decide to stay, that’s great, however, we will need to discuss the stability issues you were having, however, I’ll save that, if the issue pops up if you decide to stay.

We do appreciate your willingness to work out the issue, and I apologize for the misunderstanding, and the overzealous nature on James’ part. If you’d like to discuss any other matters, please let me know at xxx@dreamhost.com, and I’ll be happy to speak with you.

I’m going to send him an email back about using imgfly-static.feifei.us as a cache node, because I really would like to use all my bandwidth up. Everything else is moving off, however.

Post-final Update:

I had thought that was the end of it, but I noticed this comment allegedly from Dreamhost’s Co-Founder and CTO, which includes the false statement:

3. He ignored our requests to not re-enable his website and did it anyway.

I tried to leave a comment explaining that I disabled my website per instructions, then put up an index page–which was somehow misconstrued by Dreamhost as ignoring their instructions. However, it’s been a few days and my comment has not been approved. Later comments have. I guess Dreamhost feels that they should have the right to slander me on their forum. I don’t feel the same way. If any Dreamhosters want to comment here, be my guest.

New News on Dreamhost

Recently 3,500 ftp accounts got hacked, including several high-profile websites like Cameron Moll. Still, there’s nothing on their official blog about this, like they want to cover it all up.

Hi, my Firefox (the latest public version 2.0.0.1) is leaking memory. I know that you think you’ve heard this before and that it’s extensions, or old version, but seriously this has to stop:

Fixed! Removing the Firebug extension completely solves the problem. I don’t know why Firebug leaks memory, but trust me, it does.

Yup, it’s using 623MB of memory. Opening a new tab is visibly sluggish. Closing a tab, clicking on links–every action takes seconds to perform on my Core Duo 2 6600 processor with 4 GB of RAM. Extensions? I’m running two: Firebug and an S3 attachment:

The secret Firefox memory cache page (about:cache) returns nothing out of the ordinary:

Number of entries: 1114
Maximum storage size: 28672 KiB
Storage in use: 75316 KiB
Inactive storage: 0 KiB

I have no idea what’s causing this behavior, so I’m going to ask for help. Digg this and let the world know Firefox *still* has memory management issues.

Media Temple has an excellent article on their blog about the abuse of their poor MySQL servers on the “grid”. Their problems stemmed from a new user-base who had poorly performant web-applications:

Having an 8-year history of catering to high-demand websites we thought we had seen it all but this new level of load “requirement” was blindsiding. Our new offering quickly became a refuge for sites that were kicked off their old hosting company; a common industry practice. Because of their high database load “requirements” and need of resources, these site owners were shut down immediately and told to leave other hosts. Many of these “orphaned” users had applications, code, and query instructions that were grossly inefficient for even a massive dedicated server.

This is something sites like Wordpress.com don’t have to worry about, because they’re running their own code. They know they’ll scale by adding more hardware. That’s something harder to say when an idiot who’s hacked their blog software to do 1000s of cartesian joins suddenly connected to a mysql cluster node and takes it down. And if he fails over to another one…