Details

Created: 26 September 2013

If you need to parse HTTP headers from a pcap file, this is a straight forward way to do it.
I saw a lot of hacks on how to do the same thing but, in my opinion, this is the easiest way.

Tshark command

tshark -V -O http -r yourfile.pcap -R "tcp && (http.response || http.request)"

Example Perl script

$result = `tshark -V -O http -r yourfile.pcap -R "tcp && (http.response || http.request)"`;
@chunks = split ("\n\n", $result);

foreach $chunk (@chunks) {
        if ($chunk =~ /^Frame (\d+?):.*Hypertext Transfer Protocol\n(.*)\\r\\n\s+\\r\\n/s) {
                print "## Frame: $1 ##\n$2\n###############\n\n";
        }
}

Details

Created: 04 September 2013

This tutorial is to set up an Ubuntu machine to do SSL man-in-the-middle. The target for SSL interception will be a Windows machine.
Prerequisites:

# apt-get install python-dev
# apt-get install python-pip
# apt-get install libxml2-dev
# apt-get install libxslt1-dev

1. Install mitmproxy

We will be using mitmproxy for interception (mitmproxy.org)

# pip install mitmproxy

Details

Created: 15 August 2013

This tutorial is customized for an Ubuntu installation and will require the following:

# apt-get install wget
# apt-get install perl5
# apt-get install unzip

 

The following steps will get you started on building a simple web scraper in perl

1. Create a target list

You can create your own custom list or you can download a file from alexa.com containing the top 1 million visited websites

# wget http://s3.amazonaws.com/alexa-static/top-1m.csv.zip

unzip the file:

# unzip top-1m.csv.zip

The unzipped file top-1m.csv should have the following format:

1,google.com
2,facebook.com
3,youtube.com
4,yahoo.com
5,baidu.com
...

2. Perl code to read a file

In order to read a file, you need to open it and loop through each line:

open FH, "top-1m.csv";
while (my $line =<FH> ) {
  chomp $line;
  print "$line\n";
}
close FH;

3. Perl code to parse each line

Inside the while loop, use a regular expression to parse each line:

if ($line =~ /^(\d+),(.*)$/) {
  my $index = $1;
  my $website = $2;
}

4. Download webpages

Inside the if statement, call wget:

my $command = "wget -P $website/ $website";
print "$index: $command\n";
`$command`;

You may not want to scrape all 1 million websites, so you can add the following:

if ($index>100) {exit;} # this wil limit how many websites you scrape to the first 100

Details

Created: 15 August 2013

If you are experiencing a "MySQL client ran out of memory' error, it may be because you are retrieving a very large table. By default, MySQL will try to buffer all of it into memory which will cause that error.

In order to force MySQL to process each row, one at a time, set the mysql_use_result value of the db handler to 1:

$dbh= DBI->connect("DBI:mysql:<database>", "root", "<password>");
$dbh->{'mysql_use_result'}=1;