I wanted to do some basic analysis on the September 11 Pager Message dump from wikileaks. The data is about 2 years old now, and I’m sure it’s already been analysed to death but I wanted to have a play anyway.
First I binned the messages in to 5min chunks and looks for occurrences of the word “plane”. You can see the results in the graph below:
There’s an unexpected blip at around 7am, this is due to messages about a US spy drone which was shot down over Iraq before the WTC planes hit.
A lot of people initially thought it was a bomb, here are the occurrences of “bomb” in text messages:
The peak at 3pm seems to represent reports of a “Car bomb explodes outside State Department, senior law enforceement officials say” mostly from yahoo news alerts.
Next I wanted to look for server issues so I search for any message containing: offline, timeout, error (but not terror) or “not responding”:
Not sure what the peak at 5am is, there are a lot of: ” adhocdb TEXT: adhocdb unix: WARNING: vxvm:vxio: Subdisk disk02-01 block 240384: Uncorrectable write error” and “[email protected]||elgweb8 login content match error on step 2”. It appears to be mostly the later. But you can certainly see the rate of errors pick up from about 8:00 onward. To know how much of this was due to the attacks I’d need more of a baseline. Some of the more interesting ones:
2001-09-11 09:05:20 Arch [0912377] C ALPHA 8628**Customer reported outage* Internet is unavailable. Impact:User unable to connect to internet. Multiple users state either extremely slow connection or connection timed out error. Occurred:08:55
2001-09-11 09:05:31 Skytel [003920778] C ALPHA SEV1 DesMoines as PICS H0864464 SLI=Y ETA=Na 7:44 I:Pics is down, unable to access orders - error 202 no records found. E:PICS Helpdesk Janie ATCHelpdesk 512-248-4967-ATC Helpdesk
2001-09-11 09:15:46 Skytel [007607560] C ALPHA [email protected]||NSSW3/General Tire/T3149925/1-888-212-5447/Continental GT users in Charlotte are unable to connect to IBM IIN. NSCOMSOFT seeing errors. Please call ADVNETO.
Notes
I downloaded the wikileaks 911 pager torrent from: http://file.wikileaks.org/torrent/9-11_all_messages.7z.torrent. The pager messages are broken down by minute but for my analysis I wanted them all in a single file. So I started by concatenating them all.
cat 2001* > all_messages
The file contains lines that look like:
2001-09-11 03:00:00 Metrocall [1060278] B ALPHA 09/12@03:03:50 BETA13:Service (Oracle Web Lsnr 4.0(admin -DEFAULT_HOME,Intranet)) is not responding. Stopped. 1
2001-09-11 03:00:00 Metrocall [1421210] C ALPHA LAKEJob exceeded 4 hours on Lake
2001-09-11 03:00:00 Metrocall [1421210] C ALPHA LAKEJob 378304/VSIOWNER/OMSJRNMGR has MSGW status.
2001-09-11 03:00:00 Metrocall [0007690] C ALPHA THIS IS A TEST PERIODIC PAGE SEQUENTIAL NUMBER 4719
2001-09-11 03:00:01 Arch [0485957] B ALPHA (24)[email protected]|10.134.192.34 VARESAAPP03 is UP at 01:59:12
2001-09-11 03:00:01 Arch [0987275] C ALPHA s0191: 09/11 12:28:34 Reboot NT machine gblnetnt05 in cabinet 311R at 13/1CMP:CRITICAL:Sep 11 12:28:34
2001-09-11 03:00:01 Arch [1425048] C ALPHA 300~MPfetchData:openConnectionToManager:ERROR CONNECTING:192.168.35.97 : www36 connectToServerPort:socket/socket timed out at /home/crdtdrv/creditderivatives/script/MPfetchData.pl line 342, <SOCK_192.168.35.19> chunk 193021.
I then wrote some C++ code to analyse the data, here’s the “bomb” count in 5min bins across the time period:
stringify.h header:
#include <sstream>
#include <iostream>
#include <string>
#include <stdexcept>
#ifndef STRINGIFY_H
#define STRINGIFY_H
class BadConversion : public std::runtime_error {
public:
BadConversion(const std::string&amp;amp;amp;amp; s)
: std::runtime_error(s) {}
};
template<class _type>
inline std::string stringify(_type x)
{
std::ostringstream o;
if (!(o << std::fixed << x))
throw BadConversion("stringify()");
return o.str();
}
template<class _type>
inline _type convertTo(const std::string&amp;amp;amp;amp; s)
{
std::istringstream i(s);
_type x;
if (!(i >> x))
throw BadConversion("convertTo(\"" + s + "\")");
return x;
}
#endif
Analysis code:
#include
#include
#include
#include
#include “stringify.h”
#include
#include
using namespace std;
// Example line: 2001-09-11 03:00:00 Metrocall [1060278] B ALPHA 09/12@03:03:50 BETA13:Service (Oracle Web Lsnr 4.0(admin -DEFAULT_HOME,Intranet)) is not responding. Stopped. 1
int time_to_int(string date,string time) {
int seconds_per_day = 86400;
int seconds_per_hour = 3600;
int seconds_per_min = 60;
int day = convertTo(date.substr(8,2));
int hour = convertTo(time.substr(0,2));
int min = convertTo(time.substr(3,2));
int sec = convertTo(time.substr(6,2));
int itime = 0;
if(day == 12) itime += seconds_per_day;
itime += hour * seconds_per_hour;
itime += min * seconds_per_min;
itime += sec;
return itime;
}
int main() {
vector > message_bins;
int bin_size = 5; // mins
ifstream data_file(“all_messages”);
int start_time;
int next_time;
bool first=true;
vector current_bin;
for(;!data_file.eof();) {
char in_line[10000];
data_file.getline(in_line,10000);
if(data_file.eof()) break;
string line(in_line);
istringstream ss(line);
string date;
string time;
ss >> date;
ss >> time;
int i_time = time_to_int(date,time);
if(first) { start_time = i_time; next_time = start_time + 60*bin_size; first=false;}
if(i_time >= next_time) {
message_bins.push_back(current_bin);
current_bin.clear();
next_time += 60*bin_size;
}
std::transform(line.begin(), line.end(), line.begin(), ::tolower);
current_bin.push_back(line);
}
for(size_t n=0;n
RSS