The Site With the Lamp

May 11, 2005

For the third day running, the topic is log analysis tools. Earlier I wrote that webalizer/webdruid were unsuitable for my requirement because they refused to process my logfiles properly. It turns out that the real issue is not the logformat but the sequence.

In whodunnits the best suspect usually comes out smelling of roses in the end. When a web log analyzer does not produce the expected result the fingure is usually pointed at the logformat and jump to the conclusion that it is not being detected properly. In some tools the order of the log entries is more important.

For example if the tools first sees a record for May 1 and then subsequently sees a record for for April 30, it will choke. Fortunately there is a solution.

Awstats comes to the rescue of webalizer and Web Druid. It (awstats) contains a script named logresolvemerge.pl that will sort the logs into the proper order. The output produced can be safely passed into either webdruid or webalizer (and even awstats) itselfs and the files will be properly processed.

Another drawback of webalizer and it's fork webdruid is that you cannot set any filters for status code. For example in analog, you can produce a report for all redirection requests (usefull if you have restructured your site). The web* combo does not produce such a result and does not even display the redirected requests properly in the all urls page. To overcome this, I made a slight mod to the webdruid.c file. Let me know if you want this updated version.