$Id: README,v 1.3 2011/10/01 18:42:47 rdilley Exp $ === Templater (tmpltr) by Ron Dilley For the latest information on tmpltr, please see: http://www.uberadmin.com/Projects/tmpltr/ == What is Templater (tmpltr)? Templater is a small and fast log processor that provides simple artificial ignorance capabilities. You use the tool to process past log data and store templates that represent normal log line structures. You then run the tool against current or target logs and all normal patterns are automatically ignored. The parser is fast and capable of processing millions of lines per minute. For weblogs and firewall logs, I average 9M lines per minute on a 2GHz x86 machine running *NIX. The template strategy was originally proposed by a friend of mine in 2003 who later built a tool called never before seen (NBS*) which also provides artificial ignorance for arbitrary text data as well as text structures. * http://www.ranum.com/security/computer_security/code/nbs.tar == Why use it? I built this tool to solve a log analysis problem that I have suffered through while responding to many security breaches. Invariably, I need to find a needle in a haystack of log data. If you need to find a pattern that has not occurred previously, then this is the tool for you. == Implementation Templater has a simple command lines interface. In it's simplest form, pass a text file as an argument and the output will be list of counts and unique templates with the first full log line separated by a '||'. Including the first full log line with each unique template allows for quick recognition of the log lines that map to the unique template. To get a list of all the options, you can execute the command with the -h or --help switch. ---- $ ./src/tmpltr --help tmpltr v0.3 [Sep 23 2011 - 12:24:52] syntax: tmpltr [options] -d|--debug (0-9) enable debugging info -h|--help this info -t|--templates {file} load templates to ignore -v|--version display version information -w|--write {file} save templates to file The debug option is most useful when the tool is compiled with the --ENABLE-DEBUG switch. ---- A typical run of tmpltr is to pass the target log file as an argument and send the output through 'sort -n' to produce the following sorted list of unique templates and their frequency of occurent from least to most prevalent. I normally go through the list in this order to speed up catching the anonlylous patterns. ---- % tmpltr /var/log/system.log | sort -n Opening [/var/log/system.log] for read 1 %s %d %d:%d:%d %s %s %s[%d]: %s: %s %s %d %s:||Sep 23 12:11:32 MacBook Software Update[3938]: SWU: scan found 0 products: 1 %s %d %d:%d:%d %s %s[%d] (%s[%d]): %s %s %s %s: %d||Sep 23 12:11:51 MacBook com.apple.launchd[1] (com.apple.suhelperd[3940]): Exited with exit code: 2 2 %s %d %d:%d:%d %s %s[%d]: %s %s %s "%s"||Sep 23 11:52:29 macbook configd[13]: setting hostname to "macbook" 2 %s %d %d:%d:%d %s %s[%d]: %s %s -%d.%d %c||Sep 23 12:16:56 MacBook ntpd[24]: time reset -11.988784 s 2 %s %d %d:%d:%d %s %s[%d]: %s %s, -%d||Sep 23 10:27:39 MacBook eapolclient[3884]: SecKeychainFindGenericPassword failed, -25300 2 %s %d %d:%d:%d %s %s[%d]: %s: %s %s %s %s %s %s||Sep 23 10:27:39 MacBook eapolclient[3884]: en0: failed to retrieve password from keychain 3 %s %d %d:%d:%d %s [%s].%s[%d]: %s %s, %s=-%d||Sep 22 07:39:36 MacBook [0x0-0xd00d].com.hp.HPEventHandler[148]: on write, error=-1 4 %s %d %d:%d:%d %s %s[%d]: %s: %s %s||Sep 23 12:09:25 MacBook eapolclient[3919]: eapmschapv2_success_request: successfully authenticated 6 %s %d %d:%d:%d %s %s[%d]: %s: %s %s %s %s %s (%d.%d.%d.%d)||Sep 22 22:33:47 MacBook mDNSResponder[35]: RegisterInterface: Frequent transitions for interface en0 (172.20.1.165) 6 %s %d %d:%d:%d %s [%s].%s[%d]: %s %s %s %s %s %s %s - %s %s %s %s %s %s %s||Sep 22 07:39:50 MacBook [0x0-0x50050].backupd-helper[3428]: Not starting Time Machine backup after wake - failed to resolve alias to backup volume 7 %s %d %d:%d:%d %s %s[%d]: %s %s %s %s %s %s||Sep 22 07:39:50 MacBook loginwindow[36]: no spins reported for this wake 8 %s %d %d:%d:%d %s %s[%d]: %s %s %s %s %s %d %s||Sep 22 06:38:40 MacBook configd[13]: PMConnection AirPort configd plug-in com.apple.powermanagement.applicationresponse.slowresponse 1089 ms 8 %s %d %d:%d:%d %s %s[%d]: %s %s %s %s %s, %s:%d/%d/%d %d:%d:%d.%d %s - %s:%d/%d/%d %d:%d:%d.%d %s = %s:%d.%d||Sep 22 07:39:20 MacBook loginwindow[36]: loginwindow SleepWakeCallback will power on, Currenttime:9/22/2011 7:39:20.014 AM - Waketime:9/22/2011 7:39:19.220 AM = Deltatime:0.793929040 8 %s %d %d:%d:%d %s %s[%d]: %s %s %s %s||Sep 22 01:01:43 MacBook loginwindow[36]: loginwindow SleepWakeCallback WILL sleep 9 %s %d %d:%d:%d %s %s[%d]: %s %s %s %d, %s||Sep 22 07:39:25 MacBook vmnet-bridge[167]: Started bridge for 0, en0 9 %s %d %d:%d:%d %s %s[%d]: %s %s %s %s %s %s: %s:/%s/%s/%s||Sep 22 06:38:39 MacBook vmnet-bridge[167]: Failed to read SCproperties for key: State:/Network/Global/IPv4 9 %s %d %d:%d:%d %s %s[%d]: %s %s %s: %s||Sep 22 06:38:39 MacBook vmnet-bridge[167]: Stopping bridge for: en0 9 %s %d %d:%d:%d %s [%s].%s[%d]: [%s] [%s %s] %s %s %s %s %s://%s/%s/%s (%s %s %s %s://%s %s %s)||Sep 22 00:57:29 MacBook [0x0-0x13013].org.mozilla.firefox[202]: [NoScript] [NoScript ClearClick] Swallowed event keyup on chrome://browser/content/browser.xul (rapid fire from http://www.uberadmin.com in 400ms) 10 %s %d %d:%d:%d %s %s[%d]: %s: %s - %s %d - %s %s||Sep 22 07:39:21 MacBook configd[13]: Sleep: Success - BATT 99 - Maintenance Sleep 10 %s %d %d:%d:%d %s %s[%d]: %s: %s - %s %d - %s||Sep 22 07:39:21 MacBook configd[13]: Wake: Success - BATT 99 - EC.LidOpen 10 %s %d %d:%d:%d: --- %s %s %s %d %s ---||Sep 22 07:39:20: --- last message repeated 1 time --- 13 %s %d %d:%d:%d %s %s[%d]: %s %s||Sep 22 07:39:21 MacBook configd[13]: Hibernate Statistics 16 %s %d %d:%d:%d %s %s[%d]: %s %s %s %d %s||Sep 22 06:38:39 MacBook configd[13]: PMConnection mDNSResponder com.apple.powermanagement.applicationresponse.slowresponse 144 ms 58 %s %d %d:%d:%d %s %s[%d]: %s %s %s||Sep 22 00:31:07 MacBook newsyslog[3326]: logfile turned over ---- You can use the same syntax but also store copies of all of the unique templates by using the '-w {fname}. When you do this, the file references with the switch is over written with a delimited list of the templates. The template file can be used with the '-t {fname}' switch to provide artificial ignorance capabilities. ---- % tmpltr -w ignore.templates /var/log/system.log | sort -n ---- After the above run of the tool, a file named ignore.templates is created in the current directory as shown below: ---- $ cat ignore.templates %s %d %d:%d:%d %s [%s].%s[%d]: [%s] [%s %s] %s %s %s %s %s://%s/%s/%s (%s %s %s %s://%s %s %s) %s %d %d:%d:%d %s %s[%d]: %s %s %s "%s" %s %d %d:%d:%d %s %s %s[%d]: %s: %s %s %d %s: %s %d %d:%d:%d %s %s[%d]: %s: %s - %s %d - %s %s %s %d %d:%d:%d %s %s[%d]: %s %s %s %s %s %s - %s %s %s %s %s %s %d %d:%d:%d %s %s[%d]: %s %s %s %d %d:%d:%d %s %s[%d]: %s(%d.%d.%d.%d) (%s=%d): %s %s %s %s %s %d %d:%d:%d %s %s[%d]: %s %s %s/%s %s %s %s %s %s %s %d %d %s %d:%d:%d, %s %s %s %s %s %d %d %s %d:%d:%d %s %d %d:%d:%d %s %s[%d]: %s %s %s %d, %s %s %d %d:%d:%d %s %s[%d] (%s[%d]): %s %s %s %s: %d %s %d %d:%d:%d %s %s[%d]: %s %s %s %s %s %s %s %d %d:%d:%d %s [%s].%s[%d]: %s %s, %s=-%d %s %d %d:%d:%d %s %s[%d]: %s %s %s %s %s %s %s %s: /%s/%s/%s/%d.%d.%d.%s %s %d %d:%d:%d %s %s[%d]: %s %s %s %s (%s: %d) - %s %s:%d.%d, # %s %s: %d, # %s %s %s: %d, %s %s: %s, %s %s %s: %d, %s %s %s: %d %s %d %d:%d:%d %s %s[%d]: %s %s %s %s %s %d %s %s %d %d:%d:%d %s %s[%d]: %s %s %s %s %s %s: %s:/%s/%s/%s %s %d %d:%d:%d %s %s[%d]: %s %s %s %d %s %s %d %d:%d:%d %s %s[%d]: %s %s -%d.%d %c %s %d %d:%d:%d %s %s[%d]: %s: %s %s %s %d %d:%d:%d %s %s[%d] (%s): %s %s: %s %s %s %d %s %s %d %d:%d:%d %s %s[%d]: _%s: %s %s %s %s '%s' %s %d %d:%d:%d %s %s[%d]: %s: %s %s %s %s %s (%d.%d.%d.%d) %s %d %d:%d:%d %s %s[%d]: %s %s %s %s %s, %s:%d/%d/%d %d:%d:%d.%d %s - %s:%d/%d/%d %d:%d:%d.%d %s = %s:%d.%d %s %d %d:%d:%d %s %s[%d]: %s: %s %s %s %s %d %s %d %d:%d:%d %s %s[%d]: %s: %s - %s %d - %s %s %d %d:%d:%d %s [%s].%s[%d]: [%s] [%s %s] %s %s %s %s %s://%s/%s/%s (%s %s %s %s: %s %s) %s %d %d:%d:%d %s %s %s %s[%d]: %s %s %s %s (%s: %d) - %s %s:%d.%d, # %s %s: %d, # %s %s %s: %d, %s %s: %s, %s %s %s: %d, %s %s %s: %d %s %d %d:%d:%d %s %s[%d]: %s %s %s %s %d %d:%d:%d %s %s[%d]: %s(%d.%d.%d.%d) (%s=%d): %s %s %s %s %d %d:%d:%d %s [%s].%s[%d]: %s %s %s %s %s %s %s - %s %s %s %s %s %s %s %s %d %d:%d:%d %s %s[%d]: %s %s %s: %s %s %d %d:%d:%d %s %s[%d]: %s %s, -%d %s %d %d:%d:%d: --- %s %s %s %d %s --- %s %d %d:%d:%d %s %s[%d]: %s %s %s %s %s %d %d:%d:%d %s %s[%d]: %s: %s %s %s %s %s %s ---- You can now pass this file back into tmplptr and all log lines that match an existing template pattern will be ignored. This is the real utility of the tool. Using this teqnique, you can train tmpltr to ignore all log patterns that have been seen before and only show log templates for new patterns. This is how I use to tool during incident repsonse. I push all the log data that I have for the period of my choice before the detected event into tmpltr and save the pattern file, then run tmpltr against the log data for the period around when the detected event happened. ---- $ ./tmpltr -t ignore.templates /var/log/system.log ---- A couple of things to note about the '-w' and '-t' switches: If you use the '-w' switch in conjunction with the '-t', you should not read from and write to the same file. This version of tmpltr would allow you to do it without an issue if you pass the read '-t {fname}' arguments before the write '-w {fname}' arguments but future versions may not work as expected. Additionally, when you load templates using the '-t' switch then write all the templates using the '-w' switch, then new file will contain all of the templates loaded from the template file as well as any new unique templates discovered in the processing of the log files. In this way, i you can have your most recent saved template file always contain all of the templates that have been discovered. == Security Implications Assume that there are errors in the tmpltr source that would allow a specially crafted logs to allow an attacker to exploit tmpltr to gain access to the computer that it is running on!!! Don't trust this software and install and use it at your own risk. == Bugs I am not a programmer by any stretch of the imagination. I have attempted to remove the obvious bugs and other programmer related errors but please keep in mind the first sentence. If you find an issue with code, please send me an e-mail with details and I will be happy to look into it. Ron Dilley ron.dilley@uberadmin.com