Techcrunch notes that AOL has released a file containing 20,000,000 queries from “anonymized” users. However, this is a problem because anything those users typed into AOL search–social security numbers, names, drug deals, etc can be cross-correlated to expose their identities. Imagine a politician ego-searching then browsing asian pornography? The scandal would just be beginning.

AOL smartly took down the download link, but once released on the web, it will always be on the web. To that end, we’re hosting the data here on our bandwidth-limited downloads platform: AOL-data.tgz. If you get in, you should get a decently fast speed.

According to Adam D’Angelo, the reason AOL published the data was for recognition in the search-engine research arena:

This was not a leak – it was intentional. In their desperation to gain recognition from the research community, AOL decided they would compromise their integrity to provide a data set that might become often-cited in research papers: “Please reference the following publication when using this collection: G. Pass, A. Chowdhury, C. Torgeson, ‘A Picture of Search’ The First International Conference on Scalable Information Systems, Hong Kong, June, 2006.” is the message before the download.

Here’s a breakdown of the core facts:

• 20,000,000 queries from 650,000 users in 2GB uncompressed tab-delimited files
• Uncensored queries for three months of AOL search service, spring 2006
• Essentially public domain
• Contains dangerous private information

Update

The data is rife with all kinds of personally identifiable data. For example, a quick grep for credit-card patterns produces the following:

grep -i -e “[0-9]\{4\}-[0-9]\{4\}-[0-9]\{4\}-[0-9]\{4\}” *.txt

• 9006-0512-xxxx-xxx
• 1550-0905-xxxx-xxxx

Looking for Social Security Numbers (SSN) turns up this HUGE amount of data:

grep -i -e “\b[0-9]\{3\}-[0-9]\{2\}-[0-9]\{4\}\b” *.txt

• kristy nicole vega hammond la. social secruity number 437-67-xxxx birth date 03 08 xx drivers license number la. 00765xxxx address 41178 rene dr. hammond la.
• pamela button 079-60-xxxx
• thomas j finney socsec 370-40-xxxx
• 419-94-xxxx thomas black
• 458-87-xxxx seguro social
• social security number 545-29-xxxx
• ssn 436-47-xxxx

I’ve censored the personal information, but there are about 200 entries of social security numbers in the test data. Searching for things that look email addresses ([a-zA-Z0-9_\-]*@[a-zA-Z0-9_\-]*\.) turns up another 60 or so.

Update 2:

If you want to get this data into a more usable form, say MySQL, try this (note that we’re not going to bother storing duplicate queries, but you might want to):

mysql> CREATE TABLE aoldata (anonid int unsigned not null, query varchar(255), querytime datetime, itemrank int unsigned, clickurl varchar(255), PRIMARY KEY(anonid, query))

Then you just need to import it, as appropriate:

LOAD DATA LOCAL INFILE ‘user-ct-test-collection-01.txt’
INTO TABLE aoldata
FIELDS TERMINATED BY ‘\t’
LINES TERMINATED BY ‘\n’
(anonid, query, querytime, itemrank, clickurl);

Other Blogs

Paul notes that the AOL data is really Google data, since AOL search is rebranded Google. Zoli has the post that started it all.

This entry was posted on Monday, August 7th, 2006 at 3:50 am and is tagged with social security numbers, asian pornography, search engine research, core facts, card patterns, research arena, service spring, drug deals, d angelo, query data, grep, aol, kristy, search service, search query, desperation, research papers, birth date, public domain, politician. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback.

29 Responses to “AOL Gate: Search Query Data Scandal”

1. full :: 1511 » The Pepsi Challenge of 2006? enter SearchMash from Google says:

   10/4/2006 at 3:42 pm

   [...] But this did catch my eye…hopefully not another AOL Gate! Personal information Activity within SearchMash. Some features of SearchMash may enable you to interact with search results beyond simply clicking through a result or navigating to another page of results. We may record the use of these features in a non-personally identifiable manner to evaluate their usefulness. Posted by Full 1511 | [...]

   Reply
2. AOL-Data.com says:

   9/7/2006 at 11:56 pm

   For anybody who is interested in HOW TO apply the AOL-Data files to SEO for legitimate purposes, check out http://www.aol-data.com. Also offers discussions on the ethics, and a whole lot of other links and info on SEO in general.

   Reply
3. 1uk3 says:

   9/6/2006 at 12:32 am

   And I have the whole database as do many other thosaunds of people now! :O

   Not that’ll I’ll do anything with it apart from browse through but lots of damage will be done by some!

   Reply
4. ShadoWraiths says:

   8/28/2006 at 3:35 am

   [...] In the meantime, execs were busy scrambling, backpedalling or both, while attempting to assure the public that the data had been deidentified. And just how deidentified was the data? On Wednesday, August 9, 2006, New York Times journalists Michael Barbaro and Tom Zeller Jr. introduced us to one of the many so-called deidentified individuals in their article, “A Face Is Exposed for AOL Searcher No. 4417749.” One gentleman, Elliot Bäck, found everything from credit cards to social security numbers among the so-called deidentified records. [...]

   Reply