[ic] interchange filter for logwatch

[Tom Hodder]

Hi all,

I had a few problems when a CGI variable parsing error, which I didn't
notice for a while, prevented my customers completing their orders, so I
coded up a filter for logwatch which parses the logs for interchange and
includes them in the logwatch report. (Hopefully enabling me to pick up
on any future problems quick)

I've sent the tar of the filter to http://www.interchange.rtfm.info, so
they can host the file somewhere on their site if they care to, in the
meanwhile if anyone wants it, you can email me direct.

For your information Logwatch is ... "Logwatch is a customizable log
analysis system. Logwatch parses through your system's logs for a given
period of time and creates a report analyzing areas that you specify, in
as much detail as you require. Logwatch is easy to use and will work
right out of the package on most systems."



Introduction:

This filter was created using the http service filter as a template that
was shipped with logwatch-6.0.2-2, which is the current stable logwatch
rpm available from;
http://www2.logwatch.org:81/tabs/download/

The filter depends on it understanding the dates in the logfiles, and I
have made no attempt to work with any other possible date formats other
than for my own system.
(I'm using the default settings for fc2)

Example log entries that it will work with;

/var/log/interchange/error.log
192.168.1.2 - - [03/May/2005:21:22:26 +2500] -
/cgi-bin/titl/ord/paypal_standby.html CGI mapping error: Syntax error in
GET input: =Return+to+Merchant
- - - [07/May/2005:10:20:56 +0100] - - STOP server (22193) on signal TERM
- - - [07/May/2005:10:20:58 +0100] - - Interchange V5.2.0
- - - [07/May/2005:10:20:59 +0100] - - START server (4358) (UNIX)

/var/log/interchange.log
May  8 14:42:23 vsd int1:
May  8 14:42:32 vsd int1: CGI mapping error: Syntax error in GET input:
gfreg
May  8 14:42:32 vsd int1:
May  8 14:50:12 vsd int1: STOP server (8849) on signal TERM
May  8 14:55:21 vsd int1: START server (9162) (UNIX)

If you look at your equivalent log files and the date formats differ,
then this script is going to require some changes to the $ilog_pattern
and $ierr_pattern variables in the
/etc/log.d/scripts/services/interchange file to work at all.

Installation:

This filter makes the assumption that interchange is logging errors to
/var/log/interchange/error.log
Where it expects this file to be can be changed via the LogFile
directive in /etc/log.d/logwatch/logfiles/interchange.conf file.

Also, logwatch should be installed to default /etc/log.d/, I use the
current stable rpm (6.0.2-2) at;
http://www2.logwatch.org:81/tabs/download/

1) To make interchange use the syslog, add the following lines to the
/etc/interchange.cfg file; (or change existing Syslog directives to match)
   SysLog  command  /usr/bin/logger
   SysLog  tag      int1
   SysLog  alert    local3.warn
   SysLog  warn     local3.info
   SysLog  info     local3.info
   SysLog  debug    local3.debug

If you decide to use a different "Syslog tag" value to "int1", then you
will have to change the ./conf/logfiles/interchange_messages.conf file
after you untar this filter to match the correct log entries.

2) Modify the /etc/syslog.conf file to log local3 messages to the
/var/log/interchange.log file.
Add this at the bottom;

# Log local3 stuff to Interchange log
local3.*                /var/log/interchange.log
# end of interchange syslog.conf

If the local3 facility is already being used in syslog.conf, you will
have to change it to an unused facility, eg local0 through to local7,
and change the interchange.cfg file above to match.

3) restart interchange and the syslogd

service interchange restart
service syslog restart

(I was thinking that maybe "pkill -HUP syslog" might be less intrusive/a
quicker restart and maybe "pkill -HUP interchange" as well, if anyone
has a view on that I would like to hear it)

4) download int_logwatch.tar.gz into /etc/log.d/ and untar it;
tar -zxvf int_logwatch.tar.gz

This should put the following files;
./conf/logfiles/interchange.conf
./conf/logfiles/interchange_messages.conf
./conf/services/interchange.conf
./scripts/services/interchange

into the right places.

As the Logwatch rpm installs a /etc/cron.daily link, which runs with the
logwatch values specified in /etc/log.d/logwatch.conf
and this file by default uses "Service All", the interchange filter
should be included in the next logwatch cron run, eg at 4am the next day.

ToDo:

I would like to run an hourly cron that just picks up interchange,
catalog and CGI/500 type errors, and email the results with a cron like
this;
logwatch --print --range last_hour --service interchange --detail Low

but I've not had time to put use $Detail selection for the output. I
guess you could do it anyway, but you might get quite a bit of fluff.
Though I'm not sure what Interchange is logging at a debug level.
The other possibility is to remove the
   SysLog  info     local3.info
   SysLog  debug    local3.debug
entries from the /etc/interchange.cfg to limit what is logged to
/var/log/interchange.log

I guess that interchange is duplicating messages to both error.log and
syslog, when syslog is configured using the "Syslog" directive.
Therefore I could remove the parsing of the error.log and concentrate on
treating the syslog entries more specifically.
However, if for some reason its not possible to reconfig the syslog.conf
file, then this filter will pick up from the error.log configured in the
/etc/log.d/conf/logfiles/interchange.conf LogFile directive.

Other interchange logfiles:

I could not make head nor tail of the log file in each catalog folder's
/logs directory, but the usertrack file looks like it could be parsed
for interesting events.

Caveats:

As the filter is parsing the logs, it stores the unique entries in a
perl %hash as keys, if logwatch is run over a very large number of log
entries and there are lots of unique entries to be stored as keys, this
might use a lot of memory. I'm not sure what the practical limit at
which these problems might occur, but I doubt the consequences would be
that good for the system.


-- 

Tom H