QuickParser



HOME

SCRIPTS

PROJECTS

READING

CV

View Ron Dilley's profile on LinkedIn




Quick Parser (quickparser) can be found on SourceForge.

What is quickparser?

Quick Parser is a fast, non-regex log parser. The first version includes a default template for Juniper (Netscreen) type key=value logs. I will release the full version with configurable templates some time in the future if my co-conspirator does not get the lead out and get his version done and released. By fast I am talking 2.6M lines per minute on a single processor 2.5GHz PC running linux and 950K lines per minute on a SunBlade 2000 with 1GHz processors.

The output is pseudo-XML. As my partner in crime describes it, the format is enough like XML to piss off the XML haters and far enough from the standard to piss off the XML lovers.

Why use it?

Quick parser is a handy tool to normalize firewall logs. We use it to push logs into an analysis tool called overwatch. It is really fast, which makes all the difference in the world when you are pushing days and weeks of logs into your data analysis tools.

To use Quick Parser pass your log data filename as an argument to quickparser and pipe the output to a file or let it all stream to standard out. Here is an example of the output:

<rec>
<time>Aug 31 01:00:22</time>
<NetScreen device_id>sec2-fw</NetScreen device_id>
<Root>system-notification-00257</Root>
<start_time>"2007-08-31 07:00:22"</start_time>
<duration>0</duration>
<policy_id>873861</policy_id>
<service>NETBIOS</service>
<proto>17</proto>
<src zone>Untrust</src zone>
<dst zone>DMZ1</dst zone>
<action>Deny</action>
<sent>0</sent>
<rcvd>0</rcvd>
<src>172.16.10.10</src>
<dst>10.10.10.10</dst>
<src_port>35775</src_port>
<dst_port>137</dst_port>
<session_id>0</session_id>
</rec>

What is in the works?

There is a template based version where you can define your own, or have quickparser build parsing templates for you.  Which means you can get the same parsing speeds on any arbitrary text logs.

Please report issues to webmaster@uberadmin.com
12421 hits since September 23, 2007
Last updated: 2011-10-29 @ 11:16am