Ease Your Log Analysis With BGP Ranking and logs-ranking

Raphael Vinot and I worked on a network security ranking project called BGP Ranking to track the malicious activities per Internet Service Provider (referenced with their ASN Autonomous System Number). The project is free software and can be downloaded, forked or updated at GitHub. As BGP Ranking recently reached a beta stage, we have now a nice set of data about the ranking of each Internet service provider in the world. Every day, we are trying to find new ways to use the dataset to improve our life and remove the boring work while doing network forensic.

A very common task when you are doing network forensic is to analyse huge stack of logs files. Sometime, you don't even know where to start as the volume is so important that you end up to look for some random patterns that might be suspicious. I wrote a small software called logs-ranking to prefix each line of a log file (currently only W3c (common/combined) logs files are supported) with the ASN and its BGP Ranking value. logs-ranking uses the whois interface of RIPE RIS to get the origin AS for IP address and the CIRCL BGP Ranking whois interface to get the current ranking.

To use it, you just to stream your log file and specify the log format (apache in this case).

cat ../logs/www.foo.be-access.log|  perl logs-ranking.pl -f apache >www.foo.be-access.log-ranked

and you'll get an output like this with the origin ASN and the ranking (a float value) prefixing the existing log line:

AS15169,1.00273578519859,74.125.... 
AS46664,1.00599888392857,173.242...

So now, you'll be able to sort your logs by the most suspicious entries at first (at least from the most suspicious Internet service provider):

sort -r -g -t"," -k2 www.foo.be-access.log-ranked

So this can be used to discriminate infected clients from Proxy logs that tries to reach bulletproof hoster where the malware C&C is hosted. Or infected machine on Internet trying to infect your latest web-based software… the ranking can be used for other purposes, it's just a matter of imagination.

Tags: networkforensic infosec freesoftware