Accessing the Common Crawl Dataset from the command line

April 15, 2012, 9:28 am

The common crawl dataset is a crawl of the web which has been made freely available on S3 as a public dataset. There are a couple of guides out there for accessing the common crawl dataset from Hadoop, but I wanted to take a peak at the data before analysing it.

Here’s how you do this from EC2, this doesn’t incur any charges, but doing this from an external host will.

First, fire up an instance on EC2 and login. Then:

sudo su
yum install git
git clone git://github.com/s3tools/s3cmd.git

The current version (April 2012) of s3cmd is borked for “requester pays” datasets. It needs patches as described here: http://arxiv.org/help/bulk_data_s3

The instructions basically say add the following lines to S3/S3.py:

        if self.s3.config.extra_headers:
          self.headers.update(self.s3.config.extra_headers)

after:

class S3Request(object):
    def __init__(self, s3, method_string, resource, headers, params = {}):
        self.s3 = s3
        self.headers = SortedDict(headers or {}, ignore_case = True)

Then install it:

python setup.py install
s3cmd --configure

In the AWS management console, go to the top right where your name is, select, select “Security Credentials” get your access key and secret key and enter them in to s3cmd. For the other options you can accept the defaults. You can then access the dataset.

List the bucket:

s3cmd ls --add-header=x-amz-request-payer:requester s3://aws-publicdatasets/common-crawl/crawl-002

Fetch a file:

s3cmd get --add-header=x-amz-request-payer:requester s3://aws-publicdatasets/common-crawl/crawl-002/2010/01/06/1/1262851198963_1.arc.gz

Once you’ve fetched a file you can decompress it as follows:

gunzip -c 1262851198963_1.arc.gz > text
Category: Uncategorized  |  Comment

Simple histogram in python,matplotlib (no display, write to png)

April 10, 2012, 2:51 am

Reads from a file called p, uses 10000 bins, filters out values < -10000. Sets a range of -10000 to 3500000, max value of 20. [sourcecode language="python"] #!/usr/bin/env python import numpy as np import matplotlib as mpl import matplotlib.mlab as mlab mpl.use('Agg') import matplotlib.pyplot as plt inp = open ("p","r") x = [] for line in inp.readlines(): if int(line) > -10000: x.append(int(line)) print x # the histogram of the data n, bins, patches = plt.hist(x, 10000, normed=0, facecolor='green') print bins print n # add a 'best fit' line #y = mlab.normpdf( bins, mu, sigma) #l = plt.plot(bins, y, 'r--', linewidth=1) plt.xlabel('Position') plt.ylabel('Population') plt.title('My data') #plt.axis([-10000,3500000, 0, 20]) plt.grid(True) plt.savefig('histogram.png') [/sourcecode]

Category: Uncategorized  |  Comment

Pairs in an array that sum to 15…

April 7, 2012, 8:56 pm

Messy and hacky, should use set rather than map. In reality should use a hash…

#include
#include

using namespace std;

int main() {

vector array;

array.push_back(1);
array.push_back(10);
array.push_back(5);

map exists;
for(size_t n=0;n

Category: Uncategorized  |  Comment

Select a random line from a file in a single pass

April 7, 2012, 2:42 pm 
#include <iostream>
#include <string>
#include <fstream>
#include <stdlib.h>
#include <vector>

using namespace std;

void getrandline(string filename,size_t &selected_line_no,string &selected_line) {

  ifstream file(filename.c_str());
  selected_line_no = 0;
  for(size_t n=1;!file.eof();n++) {

    string current_line;
    getline(file,current_line);

    if(file.eof()) break;

    if(rand()%n == 0) { selected_line = current_line; selected_line_no = n-1; }
  }

}

int main(int argc,char **argv) {

  srand(time(NULL));

  vector<size_t> count(100,0); // just for testing, 100 should be > size of the file...
  for(size_t n=0;n<100000;n++) {
   size_t linenum;
   string line;
   getrandline(argv[1],linenum,line);

   count[linenum]++;
   cout << linenum << " " << line << endl;
  }

  for(size_t n=0;n<count.size();n++) {
    cout << n << " " << count[n] << endl;
  }

}
Category: Uncategorized  |  Comment
« Previous Entries
Next Entries »