September 11 Pager Message Analysis

I wanted to do some basic analysis on the September 11 Pager Message dump from wikileaks. The data is about 2 years old now, and I’m sure it’s already been analysed to death but I wanted to have a play anyway.

First I binned the messages in to 5min chunks and looks for occurrences of the word “plane”. You can see the results in the graph below:

There’s an unexpected blip at around 7am, this is due to messages about a US spy drone which was shot down over Iraq before the WTC planes hit.

A lot of people initially thought it was a bomb, here are the occurrences of “bomb” in text messages:

The peak at 3pm seems to represent reports of a “Car bomb explodes outside State Department, senior law enforceement officials say” mostly from yahoo news alerts.

Next I wanted to look for server issues so I search for any message containing: offline, timeout, error (but not terror) or “not responding”:

Not sure what the peak at 5am is, there are a lot of: ” adhocdb TEXT: adhocdb unix: WARNING: vxvm:vxio: Subdisk disk02-01 block 240384: Uncorrectable write error” and “[email protected]||elgweb8 login content match error on step 2”. It appears to be mostly the later. But you can certainly see the rate of errors pick up from about 8:00 onward. To know how much of this was due to the attacks I’d need more of a baseline. Some of the more interesting ones:

2001-09-11 09:05:20 Arch [0912377] C  ALPHA  8628**Customer reported outage* Internet is unavailable. Impact:User unable to connect to internet.  Multiple users state either extremely slow connection or connection timed out error. Occurred:08:55
2001-09-11 09:05:31 Skytel [003920778] C  ALPHA  SEV1 DesMoines as PICS  H0864464 SLI=Y ETA=Na 7:44  I:Pics is down, unable to access orders - error 202 no records found.  E:PICS Helpdesk  Janie ATCHelpdesk 512-248-4967-ATC Helpdesk
2001-09-11 09:15:46 Skytel [007607560] C  ALPHA  [email protected]||NSSW3/General Tire/T3149925/1-888-212-5447/Continental GT users in Charlotte are unable to connect to IBM IIN. NSCOMSOFT seeing errors. Please call ADVNETO.

Notes

I downloaded the wikileaks 911 pager torrent from: http://file.wikileaks.org/torrent/9-11_all_messages.7z.torrent. The pager messages are broken down by minute but for my analysis I wanted them all in a single file. So I started by concatenating them all.

cat 2001* > all_messages

The file contains lines that look like:

2001-09-11 03:00:00 Metrocall [1060278] B ALPHA 09/12@03:03:50 BETA13:Service (Oracle Web Lsnr 4.0(admin -DEFAULT_HOME,Intranet)) is not responding. Stopped. 1
2001-09-11 03:00:00 Metrocall [1421210] C ALPHA LAKEJob exceeded 4 hours on Lake
2001-09-11 03:00:00 Metrocall [1421210] C ALPHA LAKEJob 378304/VSIOWNER/OMSJRNMGR has MSGW status.
2001-09-11 03:00:00 Metrocall [0007690] C ALPHA THIS IS A TEST PERIODIC PAGE SEQUENTIAL NUMBER 4719
2001-09-11 03:00:01 Arch [0485957] B ALPHA (24)[email protected]|10.134.192.34 VARESAAPP03 is UP at 01:59:12
2001-09-11 03:00:01 Arch [0987275] C ALPHA s0191: 09/11 12:28:34 Reboot NT machine gblnetnt05 in cabinet 311R at 13/1CMP:CRITICAL:Sep 11 12:28:34
2001-09-11 03:00:01 Arch [1425048] C ALPHA 300~MPfetchData:openConnectionToManager:ERROR CONNECTING:192.168.35.97 : www36 connectToServerPort:socket/socket timed out at /home/crdtdrv/creditderivatives/script/MPfetchData.pl line 342, <SOCK_192.168.35.19> chunk 193021.

I then wrote some C++ code to analyse the data, here’s the “bomb” count in 5min bins across the time period:

stringify.h header:

#include <sstream>
#include <iostream>
#include <string>
#include <stdexcept>

#ifndef STRINGIFY_H
#define STRINGIFY_H

class BadConversion : public std::runtime_error {
public:
 BadConversion(const std::string&amp;amp;amp;amp;amp; s)
      : std::runtime_error(s) {}
};

template<class _type>
inline std::string stringify(_type x)
{
  std::ostringstream o;
  if (!(o << std::fixed << x))
    throw BadConversion("stringify()");
  return o.str();
}

template<class _type>
inline _type convertTo(const std::string&amp;amp;amp;amp;amp; s)
{
  std::istringstream i(s);
  _type x;
  if (!(i >> x))
    throw BadConversion("convertTo(\"" + s + "\")");
  return x;
}

#endif

Analysis code:

#include
#include
#include
#include
#include “stringify.h”
#include
#include

using namespace std;

// Example line: 2001-09-11 03:00:00 Metrocall [1060278] B ALPHA 09/12@03:03:50 BETA13:Service (Oracle Web Lsnr 4.0(admin -DEFAULT_HOME,Intranet)) is not responding. Stopped. 1

int time_to_int(string date,string time) {

int seconds_per_day = 86400;
int seconds_per_hour = 3600;
int seconds_per_min = 60;

int day = convertTo(date.substr(8,2));
int hour = convertTo(time.substr(0,2));
int min = convertTo(time.substr(3,2));
int sec = convertTo(time.substr(6,2));

int itime = 0;
if(day == 12) itime += seconds_per_day;
itime += hour * seconds_per_hour;
itime += min * seconds_per_min;
itime += sec;

return itime;
}

int main() {

vector > message_bins;

int bin_size = 5; // mins

ifstream data_file(“all_messages”);

int start_time;
int next_time;

bool first=true;
vector current_bin;
for(;!data_file.eof();) {
char in_line[10000];
data_file.getline(in_line,10000);

if(data_file.eof()) break;

string line(in_line);

istringstream ss(line);

string date;
string time;
ss >> date;
ss >> time;

int i_time = time_to_int(date,time);

if(first) { start_time = i_time; next_time = start_time + 60*bin_size; first=false;}

if(i_time >= next_time) {
message_bins.push_back(current_bin);
current_bin.clear();
next_time += 60*bin_size;
}

std::transform(line.begin(), line.end(), line.begin(), ::tolower);
current_bin.push_back(line);
}

for(size_t n=0;n