QuickParser
What is quickparser?
QuickParser is a fast, non-regex log parser. The first
version includes a default template for Juniper (Netscreen) type key=value logs. I will release the full version with
configurable templates some time in the future if my
co-conspirator does not get the lead out and get his version
done and released. By fast I am talking 2.6M lines per
minute on a single processor 2.5GHz PC running linux and 950K
lines per minute on a SunBlade 2000 with 1GHz processors.
The output is pseudo-XML. As my partner in crime describes
it, the format is enough like XML to piss off the XML haters
and far enough from the standard to piss off the XML lovers.
Why use it?
Quick parser is a handy tool to normalize firewall logs. We
use it to push logs into an analysis tool called overwatch.
It is really fast, which makes all the difference in the world
when you are pushing days and weeks of logs into your data
analysis tools.
To use Quick Parser pass your log data filename as an
argument to quickparser and pipe the output to a file or
let it all stream to standard out. Here is an example of
the output:
<rec>
<time>Aug 31 01:00:22</time>
<NetScreen device_id>sec2-fw</NetScreen device_id>
<Root>system-notification-00257</Root>
<start_time>"2007-08-31 07:00:22"</start_time>
<duration>0</duration>
<policy_id>873861</policy_id>
<service>NETBIOS</service>
<proto>17</proto>
<src zone>Untrust</src zone>
<dst zone>DMZ1</dst zone>
<action>Deny</action>
<sent>0</sent>
<rcvd>0</rcvd>
<src>172.16.10.10</src>
<dst>10.10.10.10</dst>
<src_port>35775</src_port>
<dst_port>137</dst_port>
<session_id>0</session_id>
</rec>
What is in the works?
There is a template based version where you can define your own, or
have quickparser build parsing templates for you. Which means you
can get the same parsing speeds on any arbitrary text logs.
Please report issues to webmaster@uberadmin.com
02682 hits since September 23, 2007
Last updated: 2008-08-21 @ 11:57pm