{"id":30,"date":"2011-09-11T23:25:21","date_gmt":"2011-09-11T23:25:21","guid":{"rendered":"http:\/\/41j.com\/blog\/?p=30"},"modified":"2011-09-28T16:03:17","modified_gmt":"2011-09-28T16:03:17","slug":"september-11-pager-message-analysis","status":"publish","type":"post","link":"https:\/\/41j.com\/blog\/2011\/09\/september-11-pager-message-analysis\/","title":{"rendered":"September 11 Pager Message Analysis"},"content":{"rendered":"

I wanted to do some basic analysis on the September 11 Pager Message dump from wikileaks. The data is about 2 years old now, and I’m sure it’s already been analysed to death but I wanted to have a play anyway.<\/p>\n

First I binned the messages in to 5min chunks and looks for occurrences of the word “plane”. You can see the results in the graph below:<\/p>\n

\"\"<\/a><\/p>\n

There’s an unexpected blip at around 7am, this is due to messages about a US spy drone which was shot down over Iraq before the WTC planes hit.<\/p>\n

A lot of people initially thought it was a bomb, here are the occurrences of “bomb” in text messages:<\/p>\n

\"\"<\/a><\/p>\n

The peak at 3pm seems to represent reports of a “Car bomb explodes outside State Department, senior law enforceement officials say” mostly from yahoo news alerts.<\/p>\n

Next I wanted to look for server issues so I search for any message containing: offline, timeout, error (but not terror) or “not responding”:<\/p>\n

\"\"<\/a><\/p>\n

Not sure what the peak at 5am is, there are a lot of: ” adhocdb TEXT: adhocdb unix: WARNING: vxvm:vxio: Subdisk disk02-01 block 240384: Uncorrectable write error” and “fusasitescope@wingspan.com||elgweb8 login content match error on step 2”. It appears to be mostly the later. But you can certainly see the rate of errors pick up from about 8:00 onward. To know how much of this was due to the attacks I’d need more of a baseline. Some of the more interesting ones:<\/p>\n

\r\n2001-09-11 09:05:20 Arch [0912377] C  ALPHA  8628**Customer reported outage* Internet is unavailable. Impact:User unable to connect to internet.  Multiple users state either extremely slow connection or connection timed out error. Occurred:08:55\r\n2001-09-11 09:05:31 Skytel [003920778] C  ALPHA  SEV1 DesMoines as PICS  H0864464 SLI=Y ETA=Na 7:44  I:Pics is down, unable to access orders - error 202 no records found.  E:PICS Helpdesk  Janie ATCHelpdesk 512-248-4967-ATC Helpdesk\r\n2001-09-11 09:15:46 Skytel [007607560] C  ALPHA  liacm@us.ibm.com||NSSW3\/General Tire\/T3149925\/1-888-212-5447\/Continental GT users in Charlotte are unable to connect to IBM IIN. NSCOMSOFT seeing errors. Please call ADVNETO.\r\n<\/pre>\n
Notes<\/h2>\n
I downloaded the wikileaks 911 pager torrent from: http:\/\/file.wikileaks.org\/torrent\/9-11_all_messages.7z.torrent.\u00a0The pager messages are broken down by minute but for my analysis I wanted them all in a single file. So I started by concatenating them all.<\/p>\n
\r\ncat 2001* > all_messages\r\n<\/pre>\n
The file contains lines that look like:<\/p>\n
\r\n2001-09-11 03:00:00 Metrocall [1060278] B ALPHA 09\/12@03:03:50 BETA13:Service (Oracle Web Lsnr 4.0(admin -DEFAULT_HOME,Intranet)) is not responding. Stopped. 1\r\n2001-09-11 03:00:00 Metrocall [1421210] C ALPHA LAKEJob exceeded 4 hours on Lake\r\n2001-09-11 03:00:00 Metrocall [1421210] C ALPHA LAKEJob 378304\/VSIOWNER\/OMSJRNMGR has MSGW status.\r\n2001-09-11 03:00:00 Metrocall [0007690] C ALPHA THIS IS A TEST PERIODIC PAGE SEQUENTIAL NUMBER 4719\r\n2001-09-11 03:00:01 Arch [0485957] B ALPHA (24)WhatsUp@mail.spr|10.134.192.34 VARESAAPP03 is UP at 01:59:12\r\n2001-09-11 03:00:01 Arch [0987275] C ALPHA s0191: 09\/11 12:28:34 Reboot NT machine gblnetnt05 in cabinet 311R at 13\/1CMP:CRITICAL:Sep 11 12:28:34\r\n2001-09-11 03:00:01 Arch [1425048] C ALPHA 300~MPfetchData:openConnectionToManager:ERROR CONNECTING:192.168.35.97 : www36 connectToServerPort:socket\/socket timed out at \/home\/crdtdrv\/creditderivatives\/script\/MPfetchData.pl line 342, <SOCK_192.168.35.19> chunk 193021.\r\n<\/pre>\n
I then wrote some C++ code to analyse the data, here’s the “bomb” count in 5min bins across the time period:<\/p>\n
stringify.h header:<\/p>\n
\r\n#include <sstream>\r\n#include <iostream>\r\n#include <string>\r\n#include <stdexcept>\r\n\r\n#ifndef STRINGIFY_H\r\n#define STRINGIFY_H\r\n\r\nclass BadConversion : public std::runtime_error {\r\npublic:\r\n BadConversion(const std::string&amp;amp;amp;amp;amp; s)\r\n      : std::runtime_error(s) {}\r\n};\r\n\r\ntemplate<class _type>\r\ninline std::string stringify(_type x)\r\n{\r\n  std::ostringstream o;\r\n  if (!(o << std::fixed << x))\r\n    throw BadConversion("stringify()");\r\n  return o.str();\r\n}\r\n\r\ntemplate<class _type>\r\ninline _type convertTo(const std::string&amp;amp;amp;amp;amp; s)\r\n{\r\n  std::istringstream i(s);\r\n  _type x;\r\n  if (!(i >> x))\r\n    throw BadConversion("convertTo(\\"" + s + "\\")");\r\n  return x;\r\n}\r\n\r\n#endif\r\n\r\n<\/pre>\n
Analysis code:<\/p>\n
#include 
\n#include 
\n#include 
\n#include 
\n#include “stringify.h”
\n#include 
\n#include <\/p>\n
using namespace std;<\/p>\n
\/\/ Example line: 2001-09-11 03:00:00 Metrocall [1060278] B  ALPHA  09\/12@03:03:50 BETA13:Service (Oracle Web Lsnr 4.0(admin -DEFAULT_HOME,Intranet)) is not responding. Stopped. 1<\/p>\n
int time_to_int(string date,string time) {<\/p>\n
  int seconds_per_day  = 86400;
\n  int seconds_per_hour = 3600;
\n  int seconds_per_min  = 60;<\/p>\n
  int day   = convertTo(date.substr(8,2));
\n  int hour  = convertTo(time.substr(0,2));
\n  int min   = convertTo(time.substr(3,2));
\n  int sec   = convertTo(time.substr(6,2));<\/p>\n
  int itime = 0;
\n  if(day == 12) itime += seconds_per_day;
\n  itime += hour * seconds_per_hour;
\n  itime += min  * seconds_per_min;
\n  itime += sec;<\/p>\n
  return itime;
\n}<\/p>\n
int main() {<\/p>\n
  vector > message_bins;<\/p>\n
  int bin_size = 5; \/\/ mins<\/p>\n
  ifstream data_file(“all_messages”);<\/p>\n
  int start_time;
\n  int next_time;<\/p>\n
  bool first=true;
\n  vector current_bin;
\n  for(;!data_file.eof();) {
\n    char in_line[10000];
\n    data_file.getline(in_line,10000);<\/p>\n
    if(data_file.eof()) break;<\/p>\n
    string line(in_line);<\/p>\n
    istringstream ss(line);<\/p>\n
    string date;
\n    string time;
\n    ss >> date;
\n    ss >> time;<\/p>\n
    int i_time = time_to_int(date,time);<\/p>\n
    if(first) { start_time = i_time; next_time = start_time + 60*bin_size; first=false;}<\/p>\n
    if(i_time >= next_time) {
\n      message_bins.push_back(current_bin);
\n      current_bin.clear();
\n      next_time += 60*bin_size;
\n    }<\/p>\n
    std::transform(line.begin(), line.end(), line.begin(), ::tolower);
\n    current_bin.push_back(line);
\n  }<\/p>\n
  for(size_t n=0;n\n","protected":false},"excerpt":{"rendered":"
I wanted to do some basic analysis on the September 11 Pager Message dump from wikileaks. The data is about 2 years old now, and I’m sure it’s already been analysed to death but I wanted to have a play anyway. First I binned the messages in to 5min chunks and looks for occurrences of […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[1],"tags":[3,4],"class_list":["post-30","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-11-sept","tag-september-11"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p1RRoU-u","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/posts\/30","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/comments?post=30"}],"version-history":[{"count":16,"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/posts\/30\/revisions"}],"predecessor-version":[{"id":210,"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/posts\/30\/revisions\/210"}],"wp:attachment":[{"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/media?parent=30"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/categories?post=30"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/tags?post=30"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}