ApacheLogFormat Directive

Sep 8, 2015 at 6:50 PM
Hi,
I want to analyze logs from CacheGuard (www.cacheguard.com).

Two lines of my log files looks like:
192.168.110.4 prueba [07/Sep/2015:22:57:04 +0300] "POST http://vassg141.ocsp.omniroot.com/ HTTP/1.1" 200 1977 TCP_MISS HIER_DIRECT
192.168.110.4 prueba [07/Sep/2015:22:57:13 +0300] "GET http://www.gstatic.com/news-static/img/favicon.ico HTTP/1.1" 200 1519 TCP_MISS HIER_DIRECT

I've used this in the conf file:
LogType clf
ApacheLogFormat %h %u %t \"%r\" %>s %b - -

But still I'm getting "Skipping bad record for all lines"

Any ideas ?
Thanks,
Miguel
Ps.:Attached my log file
Coordinator
Sep 11, 2015 at 11:58 AM
Let me take a look. It may take a couple of days. I don't see any attachments. BTW.

Andre
Sep 11, 2015 at 12:04 PM
Hi,
Thanks.
I´ve tried also:

LogType apache
ApacheLogFormat %h %u %t \"%r\" %>s %b - -

Does not work either.

You can find the attachment here:
https://stonestepswebalizer.codeplex.com/workitem/5

Regards,
Miguel
Coordinator
Sep 12, 2015 at 3:57 PM
The issue here is that - is not a valid format character, so this format string yields six fields. Actual log records contain eight fields and are reported as invalid.

The Apache log format parser is designed to skip unknown format sequences, as long as they are formed correctly, so you can use any character that SSW currently doesn't recognize as a format character. A word of caution, though, as Apache format is expanded, these unknown characters may be used, which will break your format. For example, %- will work today.

While looking into this issue, I found a bug that crashes SSW when there are user names in the log. You can work it around now by replacing %u with %-, but for proxy logs that makes it hardly usable. At least you will be able to see the numbers and IP addresses. I will push a fix to the CodePlex repository in a couple of days, so if you are building SSW from the source, you could pick up the fix, or you can wait for the binaries, which will take a few days.
Sep 13, 2015 at 7:22 PM
Hello,
Thanks! I will wait for the windows binaries. Please let me know when those are available.
Cheers,
Miguel
Coordinator
Sep 14, 2015 at 11:51 AM
I posted v4.0.1 that fixes the issue. Give it a try.
Sep 14, 2015 at 2:55 PM
Hi,
I've downloaded the 64-bit windows version. In the download page, there's a typo it says 54 bits.

When it starts it shows me this:
Stone Steps Webalizer v4.0.1.2 (Windows 6.2.9200)
So I suppose it's the correct version
In my webalizer.conf I have:
LogType apache
ApacheLogFormat %h %u %t \"%r\" %>s %b - -
But it still tells me that is skipping all records.
Any ideas ?
Thanks,
Miguel
Coordinator
Sep 14, 2015 at 11:04 PM
Use this:
ApacheLogFormat %h %u %t \"%r\" %>s %b %- %- 
See the post above for details.
Sep 15, 2015 at 5:44 AM
Hi,
I´ve tried with this:

LogType apache
ApacheLogFormat %h %u %t \"%r\" %>s %b %- %-

But it still tells me that is skipping all records.
Any ideas ?
Thanks,
Miguel
Coordinator
Sep 15, 2015 at 11:58 AM
I can process your sample file just fine. Check the command line output and make sure it reports the correct configuration file.
Sep 15, 2015 at 12:06 PM
Hi,
I've tried both windows versions 32 and 64 bit just in case.

It´s using the correct config file, I can see this in the output:

Processed configuration file c:\Temp\webalizer-win32-4-0-1-2/webalizer.conf
Using logfile c:\Users\testuser\Downloads\rep.txt (Apache)
Processed 1253 records (1253 bad) in 0.09 seconds

Are you running linux or windows ?
Thanks,
Miguel
Coordinator
Sep 15, 2015 at 1:45 PM
Edited Sep 15, 2015 at 1:45 PM
I tested this on Windows, running the 32-bit version of SSW. Can you attach your config file? Thanks.
Sep 15, 2015 at 6:57 PM
Hi,
I've uploaded my conf file here: https://stonestepswebalizer.codeplex.com/workitem/5
Thanks,
Miguel
Coordinator
Sep 15, 2015 at 8:08 PM
Found another bug - Apache log parser wasn't initialized if DNS was disabled. Add this line to the configuration file:
DNSChildren 1
You will see an error in the output that it cannot initialize the DNS resolver. Ignore it - you will only see internal IP addresses in the logs, so it makes no difference for you. I will post a fix for this one later this week and you can revert this setting back when you get that version.
Sep 16, 2015 at 6:12 AM
Hello,
Yes, now it works. :)
Thanks!
OK, I will try the patched version, to have names instead of IPs.
Cheers,
Miguel
Coordinator
Sep 16, 2015 at 11:56 AM
Edited Sep 16, 2015 at 12:04 PM
Good. Thank you for the heads up for these bugs!

miguelpz wrote:
OK, I will try the patched version, to have names instead of IPs.
You don't have to wait for that - current version will handle DNS queries. The bug was that if DNS or GeoIP resolution was not configured, the log parser was not initialized, causing the bug. Configure more than one DNS worker (DNSchildren) for better performance. For a local DNS server you probably will do fine with four, as there's not much wait involved. Also, configure DNScache file to reduce the load on the DNS server.