Enter Cuil, a very serious competitor, packed with ex-Googlers (Tom Costello and Anna Patterson are the backbone of Cuil, and they’ve both worked at Google), and claiming to have the largest index of websites – 120 billion – in the world.

It doesn’t end there: Cuil pulls pretty much every trick in the book. Big claims about the biggest index, privacy concerns (IP addresses of users aren’t saved, making it impossible for a third party to request it from them), semi-semantic approach (Cuil’s engine recognizes the relations between certain words on a web site, which helps it rank pages better). Hell, they even pulled the energy-saving trick: the front page of Cuil is completely black, in contrast to Google’s eye-poking whiteness.

Check out the Slashdottie thread for more discussion. I’m not interested in going there; rather I’m more concerned with how relevant the results from Cuil are, compared to Google, in a stricter context of information retrieval. After all, a search engine is about finding information.

Let’s start with a query “how to rip a dvd” in Cuil and Google:

Cuil on “How To Rip a DVD”

4 of the 9 total results are spam from Ebooksbay. An additional 4 are for converting MP3s. The final result (which is quite spammy) is for ripping DVDs to a variety of formats. Score: 11%.

Google on “How To Rip a DVD”

Google gives you 7 DVD ripping guides, and three spams site of ripping software. Essentially, you have to give it a Score: 100%, since it’s pretty much the baseline in our test. Just based on what I’ve seen so far, this will be a comparison not of relative merits, but of how much less relevant the results from Cuil are compared to Google.

Cuil on “ConcurrentHashMap”

Wait, what is that in the rightmost result!!!? Yes, that winsome young woman is carefully inspecting a ConcurrentHashMap! Ahm, bad image / search results correlations aside, the search listings fail to list the authority Java documentation source (Sun’s website) and instead list 2 mirrors (java 5 and 6), 4 bug reports, 3 mailing list discussions, and 2 random libraries with a similarly named class. Score: 50%.

Google on “ConcurrentHashMap”

Google nicely gives us the Sun Java page as the first result, 2 snippets of code using this class, 6 guides to using concurrent hash maps, a benchmark, one of the same random libraries as Cuil (Oswego), and a different random library (backport-util). I’d give them Score: 80% at this task.

Anyway, I’m getting tired of writing this. Cuil just doesn’t deliver fast, consistent, high-quality search results. The relevance is quite low, in spite of the interface improvements and searching / clustering / recommendation features.

The first thing I get, as I type “Obama,” is a list of categories that SearchMe finds relevant. I click on “Politicians” and it takes me to the next screen, the main area for exploring search results:

There are a few features you should note that set the SearchMe results apart from their competition. First, they keep the list of categories you’re interested in just one click away from instant filtering at the top of the results. Second, all of the available space of the page is filled with a gigantic preview of the search results. The title of the website is shown at the bottom, along with the site URL when you mouseover the results. Essentially, their search results are a better version of Apple’s coverflow, applied to websites. Clicking on a preview will take you directly to the page of interest, in the same tab, just like most search engines do today.

Their dynamic snippets code is nice, as well, highlighting the search terms you used in multiple colours. It appears to have been implemented directly in the coverflow-like flash engine, or behind the scenes is coming back as a new layer of image, as it loads only after the high resolution preview has loaded. An unfortunate side-effect of their highlighting algorithm is that when searching for multiple words, like “Calderon de la Barca,” the words will be highlighted separately, even if found next to each other.

Not all their results work well; for example, searching for “China” leads me into irrelevance, regardless of the category I choose, and also brings up this half-rendered view of NBA China, that my own browser renders properly. Other search terms also return odd categories and funny previews, but I imagine that this is something that will improve over time. The big problems for a search engine, responsiveness and interface, are already solved as SearchMe is both lightning fast and beautiful.

If you’re interested, you can go check out their blog or signup to the private beta. Apparently, the venture is Sequoia backed, according to Techcrunch, which probably means it’s serious about being a big web search contender in the future. According to Louis Grey, the searchme spider is aggressively hitting his blog, too. It will be interesting to come back and a year and see how SearchMe has evolved. The most likely outcome for this is being acquired by one of the big four–Facebook, Google, Yahoo, or Microsoft–since it’s hard to imagine unseating any of them in the popular mindset.

But not every suggestion is a hit; most are ok, reasonable extensions, but in nearly every phrase I type in there is something hilarious. Take, for example, “learn to speak” which has the usual suspects Spanish, English, Italian, German, and the most unusual Gibberish:

If you ask me, it’s the most expressive language in the world

While most people want to learn to play a guitar, freestyle, or draw, little Johnny is learning to levitate objects:

With his mind, what else?

Americans love to sue you. They’ll do it for wrongful imprisonment, slander, or just about any kind of distress they can of, including tubal ligation failure:

I’d rather check her phone logs…

Ask’ers don’t know much about themselves, so naturally they don’t know if they are pregnant or not, as seen in this query for “how can i:”

Also, your dog is expecting!

A typo on “how do i find” led me to this gem of feng shui confusion:

Since when is this a verb?

I saved the best for last, which is the auto-complete results for “is it legal to,” a horrifying look into modern culture:

The answer is a categorical NO!

There’s an indication that this data is filtered–try starting lesbian, for example–so I’m not sure if I believe these results. My operating assumption is that this is a marketing gimmick to try and get some churn on the blogosphere and free PR. It wouldn’t be hard to drop in a single unlikely phrase to spice things up, now would it?

p.s., don’t type “how should I” or “can I sell my” or “ask me about” because they’re sophomoric. I bet Google’s users are equally idiotic, but they certainly don’t go out of their way to show that to us!

The only clear change appears to be among large scale blog networks and similar link farms, where each site in the network provides hundreds of outgoing links on each page of the blog to other blogs in the network, in some cases creating tens, even hundred of thousands of cross links. Previously such behavior has been rewarded by Google with high page rank, although it would now appear that this loop hole may now be shut.

Here’s a table of pagerank changes organized by the percent difference:

An interesting tidbit comes from Syntagma who note that “the majority of these decreases happened after a human review.” So, it might not be easy for you to fix your linking strategy and regain Pagerank automatically.

Ironically, this coincides with GOOG hitting $666 today. And, Silicon Valley is calling us “Pagerankled.” For you people out there running blogs, an immediate solution is the following:

• Make sure you nofollow any links that you don’t directly control
• Avoid using static link-farms like directories, like linking to every blog in your network from every page
• Don’t let your commenters add links to their sites

Here’s an example of the link distribution of my site after I’ve properly annotated some links with nofollow:

The green areas (header, footer, content, and some meta data) represent regular links, the red areas (advertising, sidebar links, tags, and related stories) are nofollow links, and the blue areas are dynamic links (javascript widgets) which don’t need updating. I am not sure if I want to nofollow anything else–what do you think?

Update: Forbes weighs in, “it could also be Google simply taking into account the growth of the Internet.”