[ic] Interchange Keeps Dying

Adrian P Wilkinson interchange-users@icdevgroup.org
Fri Jan 3 13:22:01 2003


Dear List,

I've had a quick skirt around the mailing list archive and can't find a
solution so I'm asking for some hints here.  After some initial problems, I
finally managed to get Interchange up and running, ran a test catalogue and
now have reinstalled it to run as the end-user's accounts rather than
'interch' and moved the data to MySQL whilst in the process.

Everything looks good and appears to be working as expected but the
Interchange server just appears to die randomly for no reason.  Sometimes
it'll die after 20 mins, sometimes after 20 hours, even when I'm still
logged into the system and there is no "idle process killer" daemon running
to my knowledge.

Here are the contents of the various log files:

/tmp/icdebug file:
Start DEBUG at Fri Jan  3 17:58:48 2003
Start DEBUG at Fri Jan  3 17:59:37 2003
Start DEBUG at Fri Jan  3 18:00:45 2003
Start DEBUG at Fri Jan  3 18:06:41 2003
[Note: I restarted the server several times to reflect configuration changes
I was testing, it's not dying every minute!]

/home/caliconn/interchange/catalogs/caliconn/error.log:
cache-ink2-bas-hsi.cableinet.co.uk E6gdfZU7:co.uk -
[03/January/2003:16:45:27 +0000] caliconn
/cgi-bin/store/admin/survey/index.html Bad data selector='session_id'
field='' key=''
[Note: Nothing to worry about, I was just playing around with surveys]

/home/caliconn/interchange/error.log:
- - - [03/January/2003:18:06:41 +0000] - - START server (22420) (INET and
UNIX)
- - - [03/January/2003:18:09:40 +0000] - - Config 'caliconn' from running
server
 (22420)
- - - [03/January/2003:18:09:40 +0000] - - Using MySQL,
DSN=dbi:mysql:interchang
e...
- - - [03/January/2003:18:09:40 +0000] - - Reconfig of caliconn successful.
[Note: Nothing here of any value]

Process list ('ps auxwww'):

USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.3  1384  468 ?        S     2002   0:17 init [3]
root         2  0.0  0.0     0    0 ?        SW    2002   0:00 [keventd]
root         3  0.0  0.0     0    0 ?        SWN   2002   0:08
[ksoftirqd_CPU0]
root         4  0.0  0.0     0    0 ?        SW    2002   1:11 [kswapd]
root         5  0.0  0.0     0    0 ?        SW    2002   0:00 [bdflush]
root         6  0.0  0.0     0    0 ?        SW    2002   0:39 [kupdated]
root         7  0.0  0.0     0    0 ?        SW<   2002   0:00 [mdrecoveryd]
root         8  0.0  0.0     0    0 ?        SW    2002   8:13 [kjournald]
root       120  0.0  0.0     0    0 ?        SW    2002   0:00 [kjournald]
root       121  0.0  0.0     0    0 ?        SW    2002   7:13 [kjournald]
root       122  0.0  0.0     0    0 ?        SW    2002  11:01 [kjournald]
root       123  0.0  0.0     0    0 ?        SW    2002  12:17 [kjournald]
root       340  0.0  0.0     0    0 ?        SW    2002   0:00 [eth0]
root       549  0.0  0.7  3232  892 ?        S     2002   0:02
/usr/sbin/sshd
root       568  0.0  0.6  2172  732 ?        S     2002   0:22
xinetd -stayalive -reuse -pidfile /var/run/xinetd.pid
root       690  0.0  0.8  7516  956 ?        S     2002   0:25
/usr/sbin/httpsd
root       713  0.0  0.4  1428  564 ?        S     2002   0:02 crond
nobody     714  0.0  0.4  4324  568 ?        S     2002   0:00
/usr/local/bin/gcache 99 /wwws/gcache_port
nobody     715  0.0  0.7  7604  944 ?        S     2002   0:00
/usr/sbin/httpsd
nobody     716  0.0  0.7  7604  944 ?        S     2002   0:00
/usr/sbin/httpsd
nobody     717  0.0  0.8  7680  948 ?        S     2002   0:00
/usr/sbin/httpsd
nobody     718  0.0  0.8  7680 1060 ?        S     2002   0:00
/usr/sbin/httpsd
nobody     719  0.0  0.7  7604  944 ?        S     2002   0:00
/usr/sbin/httpsd
nobody     720  0.0  0.7  7604  944 ?        S     2002   0:00
/usr/sbin/httpsd
nobody     721  0.0  0.7  7604  944 ?        S     2002   0:00
/usr/sbin/httpsd
nobody     723  0.0  0.8  7680  948 ?        S     2002   0:00
/usr/sbin/httpsd
nobody     724  0.0  0.7  7604  944 ?        S     2002   0:00
/usr/sbin/httpsd
root       743  0.0  1.2  4124 1496 ?        SN    2002   0:36 healthd
root       764  0.0  0.7  2220  840 ?        S     2002   0:00 /bin/sh
/usr/local/mysql/bin/safe_mysqld --datadir=/usr/local/mysql/data --pid-file=
/usr/local/mysql/data/invincible.propagation.net.pid
root       780  0.0  0.3  3780  460 ?        S     2002   0:00
/usr/local/real/Bin/rmserver /usr/local/real/rmserver.cfg --sct --iehp
root       799  0.0  3.8  7140 4592 ?        S     2002  16:06 syswatchd
mysql      852  0.0  1.0  2572 1280 ?        S     2002   0:00
/usr/local/mysql/bin/mysqld --defaults-extra-file=/usr/local/mysql/data/mysq
l.conf --basedir=/usr/local/mysql --datadir=/usr/local/mysql/data --user=mys
ql --pid-file=/usr/local/mysql/data/invincible.propagation.net.pid --skip-lo
cking
root       854  0.0  1.2 125452 1428 ?       S     2002   0:03
/usr/local/real/Bin/rmserver /usr/local/real/rmserver.cfg --sct --iehp
root       855  0.0  0.4  3780  532 ?        S     2002   0:08
/usr/local/real/Bin/rmserver /usr/local/real/rmserver.cfg --sct --iehp
root       856  0.0  1.2 125452 1428 ?       S     2002   9:27
/usr/local/real/Bin/rmserver /usr/local/real/rmserver.cfg --sct --iehp
root       870  0.0  0.2  1352  348 tty1     S     2002   0:00
/sbin/mingetty tty1
root       871  0.0  0.2  1352  348 tty2     S     2002   0:00
/sbin/mingetty tty2
root       872  0.0  0.2  1352  348 tty3     S     2002   0:00
/sbin/mingetty tty3
root       873  0.0  0.2  1352  348 tty4     S     2002   0:00
/sbin/mingetty tty4
root       874  0.0  0.2  1352  348 tty5     S     2002   0:00
/sbin/mingetty tty5
root       875  0.0  0.2  1364  348 tty6     S     2002   0:00
/sbin/agetty -i -l /sbin/elogin 9600 tty6
mysql      885  0.0  1.0  2572 1280 ?        S     2002   0:08
/usr/local/mysql/bin/mysqld --defaults-extra-file=/usr/local/mysql/data/mysq
l.conf --basedir=/usr/local/mysql --datadir=/usr/local/mysql/data --user=mys
ql --pid-file=/usr/local/mysql/data/invincible.propagation.net.pid --skip-lo
cking
mysql      886  0.0  1.0  2572 1280 ?        S     2002   0:01
/usr/local/mysql/bin/mysqld --defaults-extra-file=/usr/local/mysql/data/mysq
l.conf --basedir=/usr/local/mysql --datadir=/usr/local/mysql/data --user=mys
ql --pid-file=/usr/local/mysql/data/invincible.propagation.net.pid --skip-lo
cking
root     15151  0.0  1.2 125452 1428 ?       S     2002   1:15
/usr/local/real/Bin/rmserver /usr/local/real/rmserver.cfg --sct --iehp
root     15152  0.0  1.2 125452 1428 ?       S     2002   0:00
/usr/local/real/Bin/rmserver /usr/local/real/rmserver.cfg --sct --iehp
root     15153  0.0  1.2 125452 1428 ?       S     2002   0:00
/usr/local/real/Bin/rmserver /usr/local/real/rmserver.cfg --sct --iehp
root     15154  0.0  1.2 125452 1428 ?       S     2002   0:00
/usr/local/real/Bin/rmserver /usr/local/real/rmserver.cfg --sct --iehp
root     15155  0.0  1.2 125452 1428 ?       S     2002   0:00
/usr/local/real/Bin/rmserver /usr/local/real/rmserver.cfg --sct --iehp
root     15156  0.0  1.2 125452 1428 ?       S     2002   0:00
/usr/local/real/Bin/rmserver /usr/local/real/rmserver.cfg --sct --iehp
root     15157  0.0  1.2 125452 1428 ?       S     2002   0:00
/usr/local/real/Bin/rmserver /usr/local/real/rmserver.cfg --sct --iehp
root     15158  0.0  1.2 125452 1428 ?       S     2002   0:00
/usr/local/real/Bin/rmserver /usr/local/real/rmserver.cfg --sct --iehp
root     15159  0.0  1.2 125452 1428 ?       S     2002   0:00
/usr/local/real/Bin/rmserver /usr/local/real/rmserver.cfg --sct --iehp
root     15160  0.0  1.2 125452 1428 ?       S     2002   0:00
/usr/local/real/Bin/rmserver /usr/local/real/rmserver.cfg --sct --iehp
root     15161  0.0  1.2 125452 1428 ?       S     2002   0:00
/usr/local/real/Bin/rmserver /usr/local/real/rmserver.cfg --sct --iehp
root     15162  0.0  1.2 125452 1428 ?       S     2002   0:00
/usr/local/real/Bin/rmserver /usr/local/real/rmserver.cfg --sct --iehp
root     15163  0.0  1.2 125452 1428 ?       S     2002   0:00
/usr/local/real/Bin/rmserver /usr/local/real/rmserver.cfg --sct --iehp
root     15164  0.0  1.2 125452 1428 ?       S     2002   0:00
/usr/local/real/Bin/rmserver /usr/local/real/rmserver.cfg --sct --iehp
root     15165  0.0  1.2 125452 1428 ?       S     2002   0:00
/usr/local/real/Bin/rmserver /usr/local/real/rmserver.cfg --sct --iehp
root     15166  0.0  1.2 125452 1428 ?       S     2002   0:00
/usr/local/real/Bin/rmserver /usr/local/real/rmserver.cfg --sct --iehp
root     15167  0.0  1.2 125452 1428 ?       S     2002   0:00
/usr/local/real/Bin/rmserver /usr/local/real/rmserver.cfg --sct --iehp
root     15168  0.0  1.2 125452 1428 ?       S     2002   0:00
/usr/local/real/Bin/rmserver /usr/local/real/rmserver.cfg --sct --iehp
root     15169  0.0  1.2 125452 1428 ?       S     2002   0:00
/usr/local/real/Bin/rmserver /usr/local/real/rmserver.cfg --sct --iehp
root     15170  0.0  1.2 125452 1428 ?       S     2002   0:12
/usr/local/real/Bin/rmserver /usr/local/real/rmserver.cfg --sct --iehp
root     15171  0.0  1.2 125452 1428 ?       S     2002   0:06
/usr/local/real/Bin/rmserver /usr/local/real/rmserver.cfg --sct --iehp
root     17812  0.0  0.8  5628  956 ?        S     2002   1:14 monstd
nobody   14162  0.0  0.8  7604  956 ?        S     2002   0:00
/usr/sbin/httpsd
nobody   14164  0.0  0.8  7604  956 ?        S     2002   0:00
/usr/sbin/httpsd
nobody   14167  0.0  0.8  7604  956 ?        S     2002   0:00
/usr/sbin/httpsd
nobody   14168  0.0  0.8  7604  956 ?        S     2002   0:00
/usr/sbin/httpsd
nobody   14169  0.0  0.8  7604  956 ?        S     2002   0:00
/usr/sbin/httpsd
nobody   14170  0.0  0.8  7604  956 ?        S     2002   0:00
/usr/sbin/httpsd
nobody   14171  0.0  0.8  7604  956 ?        S     2002   0:00
/usr/sbin/httpsd
nobody   14172  0.0  0.8  7604  956 ?        S     2002   0:00
/usr/sbin/httpsd
nobody   14173  0.0  0.7  7604  932 ?        S     2002   0:00
/usr/sbin/httpsd
nobody   24825  0.0  0.8  7680 1048 ?        S     2002   0:00
/usr/sbin/httpsd
nobody   19831  0.0  0.9  7604 1068 ?        S    Jan02   0:00
/usr/sbin/httpsd
root     26165  0.0  1.1  6576 1328 ?        S    Jan02   0:01
/usr/sbin/sshd
root     26180  0.0  1.0  2412 1260 pts/0    S    Jan02   0:00 -bash
root     17206  0.0  1.3  6064 1620 ?        S    09:26   0:00
/usr/sbin/httpd
nobody   17214  0.0  1.9  6136 2260 ?        S    09:26   0:00
/usr/sbin/httpd
nobody   17215  0.0  1.9  6140 2260 ?        S    09:26   0:00
/usr/sbin/httpd
nobody   17216  0.0  1.9  6144 2264 ?        S    09:26   0:00
/usr/sbin/httpd
nobody   17217  0.0  1.9  6136 2256 ?        S    09:26   0:00
/usr/sbin/httpd
nobody   17218  0.0  1.9  6136 2260 ?        S    09:26   0:00
/usr/sbin/httpd
nobody   17219  0.0  1.9  6136 2260 ?        S    09:26   0:00
/usr/sbin/httpd
nobody   17220  0.0  1.9  6140 2264 ?        S    09:26   0:00
/usr/sbin/httpd
nobody   17221  0.0  1.9  6136 2260 ?        S    09:26   0:00
/usr/sbin/httpd
nobody   17222  0.0  1.9  6136 2260 ?        S    09:26   0:00
/usr/sbin/httpd
nobody   17223  0.0  1.9  6140 2264 ?        S    09:26   0:00
/usr/sbin/httpd
nobody   17224  0.0  1.9  6140 2264 ?        S    09:26   0:00
/usr/sbin/httpd
nobody   17225  0.0  1.9  6140 2264 ?        S    09:26   0:00
/usr/sbin/httpd
nobody   17226  0.0  1.9  6136 2260 ?        S    09:26   0:00
/usr/sbin/httpd
nobody   17227  0.0  1.9  6136 2260 ?        S    09:26   0:00
/usr/sbin/httpd
nobody   17228  0.0  1.9  6136 2260 ?        S    09:26   0:00
/usr/sbin/httpd
nobody   17229  0.0  1.9  6136 2260 ?        S    09:26   0:00
/usr/sbin/httpd
nobody   17230  0.0  1.9  6136 2260 ?        S    09:26   0:00
/usr/sbin/httpd
nobody   17231  0.0  1.9  6136 2260 ?        S    09:26   0:00
/usr/sbin/httpd
nobody   17232  0.0  1.9  6136 2260 ?        S    09:26   0:00
/usr/sbin/httpd
nobody   17233  0.0  1.9  6136 2260 ?        S    09:26   0:00
/usr/sbin/httpd
nobody   17234  0.0  1.9  6136 2260 ?        S    09:26   0:00
/usr/sbin/httpd
nobody   17235  0.0  1.9  6136 2260 ?        S    09:26   0:00
/usr/sbin/httpd
nobody   17236  0.0  1.9  6136 2260 ?        S    09:26   0:00
/usr/sbin/httpd
nobody   17237  0.0  1.9  6140 2260 ?        S    09:26   0:00
/usr/sbin/httpd
nobody   17238  0.0  1.9  6136 2260 ?        S    09:26   0:00
/usr/sbin/httpd
nobody   17239  0.0  1.9  6136 2256 ?        S    09:26   0:00
/usr/sbin/httpd
nobody   17240  0.0  1.9  6136 2256 ?        S    09:26   0:00
/usr/sbin/httpd
nobody   17241  0.0  1.9  6136 2256 ?        S    09:26   0:00
/usr/sbin/httpd
nobody   17242  0.0  1.9  6136 2256 ?        S    09:26   0:00
/usr/sbin/httpd
nobody   17243  0.0  1.9  6136 2256 ?        S    09:26   0:00
/usr/sbin/httpd
root     22195  0.0  1.5  5040 1836 ?        S    12:00   0:00 sendmail:
accepting connections
caliconn 22420  0.0 21.6 29540 25676 ?       S    12:06   0:00
/usr/local/bin/perl /home/caliconn/interchange/bin/interchange --Ignore -r
root     22825  0.0  0.4  1444  592 ?        S    12:17   0:00 syslogd -m 0
root     22830  0.0  0.9  1988 1100 ?        S    12:17   0:00 klogd -2
root     22873  0.0  0.6  2672  788 pts/0    R    12:19   0:00 ps auxwww
auxwww

Any suggestions as how I can find out -WHY- it keeps dying and how I can
stop it from doing so?

Regards, Ade.