{"id":50,"date":"2011-09-12T20:56:52","date_gmt":"2011-09-12T20:56:52","guid":{"rendered":"http:\/\/41j.com\/blog\/?p=50"},"modified":"2011-09-28T16:03:17","modified_gmt":"2011-09-28T16:03:17","slug":"aol-search-data-leak","status":"publish","type":"post","link":"https:\/\/41j.com\/blog\/2011\/09\/aol-search-data-leak\/","title":{"rendered":"AOL Search Data Leak"},"content":{"rendered":"

In 2006 there was a “leak” of AOL search data. A large set of anonymized search queries was released publicly by AOL for research purposes. AOL however quickly realised that the data was a public relations nightmare, and that some users could be deanonymized though their search queries. The dataset was removed from their website, and the incident lead to the resignation of the CTO of AOL. Anyway, the dataset is still freely available on the internet and I’m interested in playing with it a little.<\/p>\n

First of all I want to pull out the top search queries. UNIX sort and uniq handle this quite well as shown in the Notes section. Here are the top 50 queries:<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
1<\/td>\n711111<\/p>\n–<\/td>\n<\/tr>\n
2<\/td>\n233670<\/p>\ngoogle<\/td>\n<\/tr>\n
3<\/td>\n98713<\/p>\nebay<\/td>\n<\/tr>\n
4<\/td>\n91234<\/p>\nyahoo<\/td>\n<\/tr>\n
5<\/td>\n69314<\/p>\nyahoo.com<\/td>\n<\/tr>\n
6<\/td>\n61890<\/p>\nmapquest<\/td>\n<\/tr>\n
7<\/td>\n55832<\/p>\ngoogle.com<\/td>\n<\/tr>\n
8<\/td>\n54985<\/p>\nmyspace.com<\/td>\n<\/tr>\n
9<\/td>\n51433<\/p>\nmyspace<\/td>\n<\/tr>\n
10<\/td>\n30835<\/p>\nwww.yahoo.com<\/td>\n<\/tr>\n
11<\/td>\n30045<\/p>\nwww.google.com<\/td>\n<\/tr>\n
12<\/td>\n28185<\/p>\ninternet<\/td>\n<\/tr>\n
13<\/td>\n21353<\/p>\nhttp<\/td>\n<\/tr>\n
14<\/td>\n20046<\/p>\nwww.myspace.com<\/td>\n<\/tr>\n
15<\/td>\n19578<\/p>\nmap<\/td>\n<\/tr>\n
16<\/td>\n19381<\/p>\nweather<\/td>\n<\/tr>\n
17<\/td>\n18577<\/p>\nmy<\/td>\n<\/tr>\n
18<\/td>\n18560<\/p>\nebay.com<\/td>\n<\/tr>\n
19<\/td>\n16191<\/p>\nbank<\/td>\n<\/tr>\n
20<\/td>\n15940<\/p>\namerican<\/td>\n<\/tr>\n
21<\/td>\n14512<\/p>\npogo<\/td>\n<\/tr>\n
22<\/td>\n14063<\/p>\nhotmail<\/td>\n<\/tr>\n
23<\/td>\n13824<\/p>\nmsn<\/td>\n<\/tr>\n
24<\/td>\n13575<\/p>\nask<\/td>\n<\/tr>\n
25<\/td>\n13521<\/p>\ncraigslist<\/td>\n<\/tr>\n
26<\/td>\n13294<\/p>\nhotmail.com<\/td>\n<\/tr>\n
27<\/td>\n12825<\/p>\ndictionary<\/td>\n<\/tr>\n
28<\/td>\n12512<\/p>\nyahoo<\/td>\n<\/tr>\n
29<\/td>\n11877<\/p>\nmsn.com<\/td>\n<\/tr>\n
30<\/td>\n11757<\/p>\nmycl.cravelyrics.com<\/td>\n<\/tr>\n
31<\/td>\n11091<\/p>\nbankofamerica<\/td>\n<\/tr>\n
32<\/td>\n10360<\/p>\nmapquest.com<\/td>\n<\/tr>\n
33<\/td>\n10158<\/p>\nwalmart<\/td>\n<\/tr>\n
34<\/td>\n10126<\/p>\nmy<\/td>\n<\/tr>\n
35<\/td>\n10117<\/p>\nask.com<\/td>\n<\/tr>\n
36<\/td>\n9943<\/p>\ntattoo<\/td>\n<\/tr>\n
37<\/td>\n9530<\/p>\n.com<\/td>\n<\/tr>\n
38<\/td>\n9059<\/p>\nsouthwest<\/td>\n<\/tr>\n
39<\/td>\n8926<\/p>\nmyspace<\/td>\n<\/tr>\n
40<\/td>\n8906<\/p>\nwhite<\/td>\n<\/tr>\n
41<\/td>\n8510<\/p>\nmaps<\/td>\n<\/tr>\n
42<\/td>\n8342<\/p>\nsex<\/td>\n<\/tr>\n
43<\/td>\n8315<\/p>\nporn<\/td>\n<\/tr>\n
44<\/td>\n7791<\/p>\nmailbox<\/td>\n<\/tr>\n
45<\/td>\n7714<\/p>\nhome<\/td>\n<\/tr>\n
46<\/td>\n7709<\/p>\nwww.google<\/td>\n<\/tr>\n
47<\/td>\n7615<\/p>\nfidelity.com<\/td>\n<\/tr>\n
48<\/td>\n7506<\/p>\npogo.com<\/td>\n<\/tr>\n
49<\/td>\n7459<\/p>\ntarget<\/td>\n<\/tr>\n
50<\/td>\n7372<\/p>\nmatch.com<\/td>\n<\/tr>\n<\/table>\n

Notes<\/h2>\n
\r\nwget http:\/\/www.atrus.org\/hosted\/AOL-data.tgz\r\ntar xvzf AOL-data.tgz\r\nfor i in .\/user*;do sed '1 d' $i >> aol_all.txt;done\r\nawk 'BEGIN{FS="\\t";}{print $2}' aol_all.txt | sort | uniq -c | sort -n -r -k 1 > aol_all.txt.top\r\nhead aol_all.txt.top -n 50 | awk 'BEGIN{n=1;}{print "<tr><td>" n "<\/td><td>" $1 "<td>" $2 "<\/td><\/tr>";n++;}'\r\n<\/pre>\n","protected":false},"excerpt":{"rendered":"
In 2006 there was a “leak” of AOL search data. A large set of anonymized search queries was released publicly by AOL for research purposes. AOL however quickly realised that the data was a public relations nightmare, and that some users could be deanonymized though their search queries. The dataset was removed from their website, […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[1],"tags":[],"class_list":["post-50","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p1RRoU-O","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/posts\/50","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/comments?post=50"}],"version-history":[{"count":8,"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/posts\/50\/revisions"}],"predecessor-version":[{"id":209,"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/posts\/50\/revisions\/209"}],"wp:attachment":[{"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/media?parent=50"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/categories?post=50"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/tags?post=50"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}