HOME
SCRIPTS
PROJECTS
READING
CV
Quick Parser (quickparser) can be found on SourceForge.
What is quickparser?
Quick Parser is a fast, non-regex log parser. The first version includes a default template for Juniper (Netscreen) type key=value logs. I will release the full version with configurable templates some time in the future if my co-conspirator does not get the lead out and get his version done and released. By fast I am talking 2.6M lines per minute on a single processor 2.5GHz PC running linux and 950K lines per minute on a SunBlade 2000 with 1GHz processors.The output is pseudo-XML. As my partner in crime describes it, the format is enough like XML to piss off the XML haters and far enough from the standard to piss off the XML lovers.
Why use it?
Quick parser is a handy tool to normalize firewall logs. We use it to push logs into an analysis tool called overwatch. It is really fast, which makes all the difference in the world when you are pushing days and weeks of logs into your data analysis tools.To use Quick Parser pass your log data filename as an argument to quickparser and pipe the output to a file or let it all stream to standard out. Here is an example of the output:
<rec>
<time>Aug 31 01:00:22</time>
<NetScreen device_id>sec2-fw</NetScreen device_id>
<Root>system-notification-00257</Root>
<start_time>"2007-08-31 07:00:22"</start_time>
<duration>0</duration>
<policy_id>873861</policy_id>
<service>NETBIOS</service>
<proto>17</proto>
<src zone>Untrust</src zone>
<dst zone>DMZ1</dst zone>
<action>Deny</action>
<sent>0</sent>
<rcvd>0</rcvd>
<src>172.16.10.10</src>
<dst>10.10.10.10</dst>
<src_port>35775</src_port>
<dst_port>137</dst_port>
<session_id>0</session_id>
</rec>