[ic] mod_interchange and Apache MaxClients

[Ron Phipps]

> From: interchange-users-bounces at icdevgroup.org
[mailto:interchange-users-
> bounces at icdevgroup.org] On Behalf Of John1
> Sent: Tuesday, November 15, 2005 3:19 AM
> 
> On Tuesday, November 15, 2005 2:16 AM, rphipps at reliant-solutions.com
> wrote:
> 
> >> From: interchange-users-bounces at icdevgroup.org
> >> [mailto:interchange-users- bounces at icdevgroup.org] On Behalf Of
John1
> >> Sent: Monday, November 14, 2005 3:52 PM
> >>
> >> I am running Apache with mod_interchange 1.32, with the following
> >> httpd.conf
> >> entries:
> >>
> >> StartServers  5
> >> MinSpareServers  5
> >> MaxSpareServers  10
> >> MaxClients  150
> >> MaxRequestsPerChild 10000
> >>
> >> I am finding that even with these settings Apache is hitting its
> >> MaxClients limit now and again as evidenced by the following
> >> entry in the Apache error log:
> >>
> >> "server reached MaxClients setting, consider raising the MaxClients
> >> setting" The problem is that once Apache bumps up against this
limit,
> >> the site hangs, permanently!
> >>
> >> i.e.  The Apache processes running stay at 150 indefinitely and so
> >> the site becomes permanently inaccessible.
> >>
> >> I had thought that once MaxClients was reached TCP requests would
be
> >> queued and so eventually the lockout situation would end once the
> >>backlog of requests was cleared.
> >>
> >> Having come across an old post to the mailing list:
> >> http://www.icdevgroup.org/pipermail/interchange-users/2003-
> >> March/032292.html
> >>
> >> ...I am wondering whether the reason that the lockout never clears
> >> may be to do with how mod_interchange queues requests for Apache?
> >>  Does anyone know whether this could be the case, or if not, can
anyone
> >> else
> >> suggest why I am getting *permanent* lockout once the MaxClient
limit
> >> is bumped up against. i.e. currently I am forced to restart Apache
once
> >> the
> >> MaxClient limit is bumped.
> >>
> >> I have a second related question:
> >>
> >> The reason that the MaxClient is being bumped from time to time is
> >> often (though not always) due to http floods from scripts looking
> >> for files on my server that don't exist (i.e. looking for files
with
> >> known
> >> security holes).
> >>
> >> Does anyone have any suggestions on how best to stop these sort of
> >> floods bringing down my server?
> >>
> >> Any suggestions on how to solve this issue (and the lockout
problem)
> >> would be greatly appreciated as I am currently having to restart
> >> Apache several times a day.  Thanks.
> >>
> >
> > I'm hoping Kevin can elaborate on how mod_interchange interacts with
> > Apache and IC and perhaps we can come to a conclusion.  Another idea
I
> > had was that maybe the MaxClients being reached is the result of
> > another problem, such as IC is actually no longer processing the
> > requests so Apache is never able to clear its backlog.  The restart
> > of Apache, could be making IC kill off the children processes and
> > then spawning new processes in effect allowing IC to "catch up" with
> > the requests.
> >
> Yes, I was wondering about this sort of possibility too. I had
previously
> thought that it was necessary to restart Apache and Interchange to
clear
> the
> problem, but I have now found that as long as I leave the server for a
few
> seconds between stopping and restarting Apache, there is no need to
> restart
> Interchange.  Conversely, if I stop and then quickly restart Apache
> without
> pausing the website is still unresponsive.  It seems that the reason
for
> this is that once Apache has been stopped it takes a few seconds for
the
> old
> Interchange daemon processes to die off, after which Apache can be
> restarted
> and everything is fine again.  I think it is necessary to wait for all
the
> Interchange processes to die off before restarting Apache, but I can't
be
> 100% sure of this - I will try to verify whether this is definitely
the
> case
> next time I get a lockup.

Last night I recompiled with MaxClients to 1024 and this morning I ran
into the problem again, also MaxClients was not reached this time,
however I was unable to reach the IC site.  At least in my case I do not
believe MaxClients to be the issue.

When did this problem start happening for you?  How often are you seeing
this problem?  The next time the problem occurs could you restart
Interchange instead of Apache and see what happens?  I'm still trying to
figure out if this is an issue with IC or an issue with Apache.
Restarting Apache could kill off the IC children processes that were
attached to Apache and restarting IC could kill off the Apache children
that were attached to IC, so not sure quite where the issue is at.

> > When the site goes down, further requests for IC pages show the
> > following in the apache error log for the particular site:
> >
> > [Thu Nov 10 20:15:11 2005] [error] [client x.x.x.x] Malformed header
> > return by Interchange:
> > [Thu Nov 10 20:15:11 2005] [error] [client x.x.x.x] Premature end of
> > script headers: /home/xxxxxx/public_html/favicon.ico
> > [Thu Nov 10 20:15:11 2005] [error] [client x.x.x.x] Malformed header
> > return by Interchange:
> > [Thu Nov 10 20:15:14 2005] [error] [client x.x.x.x] Premature end of
> > script headers: /home/xxxxxx/public_html/index.html
> > [Thu Nov 10 20:15:14 2005] [error] [client x.x.x.x] Malformed header
> > return by Interchange:
> > [Thu Nov 10 20:15:16 2005] [error] [client x.x.x.x] Premature end of
> > script headers: /home/xxxxxx/public_html/index.html
> > [Thu Nov 10 20:15:16 2005] [error] [client x.x.x.x] Malformed header
> > return by Interchange:
> > [Thu Nov 10 20:15:18 2005] [error] [client x.x.x.x] Premature end of
> > script headers: /home/xxxxxx/public_html/scan
> > [Thu Nov 10 20:15:18 2005] [error] [client x.x.x.x] Malformed header
> > return by Interchange:
> > [Thu Nov 10 20:15:20 2005] [error] [client x.x.x.x] Premature end of
> > script headers: /home/xxxxxx/public_html/favicon.ico
> > [Thu Nov 10 20:15:20 2005] [error] [client x.x.x.x] Malformed header
> > return by Interchange:
> > [Thu Nov 10 20:15:21 2005] [error] [client x.x.x.x] Premature end of
> > script headers: /home/xxxxxx/public_html/robots.txt
> > [Thu Nov 10 20:15:21 2005] [error] [client x.x.x.x] Malformed header
> > return by Interchange:
> > [Thu Nov 10 20:15:24 2005] [error] [client x.x.x.x Premature end of
> > script headers: /home/xxxxxx/public_html/index.html
> > [Thu Nov 10 20:15:24 2005] [error] [client x.x.x.x] Malformed header
> > return by Interchange:
> >
> Incidentally, I have another very low traffic, non-Interchange website
> running on the same server.  I have noticed that even if these "script
> attacks" are against the other non-Interchange website, they still
cause
> the
> permanent lockup of Apache and hence the Interchange site once
MaxClients
> is
> hit.
> 
> > These messages repeat until IC is restarted.
> >
> > We've noticed more scripts hitting our site looking for holes and a
> > higher amount of search engines crawling the site so higer traffic
> > patterns may be related.

Do you see those same error messages when your site goes down for
traffic after the site goes down?

> > John and Jeff, could you guys post the following:
> >
> > OS version
> > Apache version
> > Mysql version
> > Perl version
> > Mod_interchange version
> > IC version
> > Running in RPC mode?
> > Processors?
> > Ram?
> >
> I am running an Interchange 5.3.0 nightly build (that I have found to
be
> otherwise stable for over a year), mod_interchange 1.32.  Interchange
is
> running in RPC mode.  It is running on a Virtuozzo Virtual server
platform
> so processors and RAM probably not that relevant. Perl Version 5.6.1
and
> MySql version 4.1.6.
> 

So it appears that this problem is not dependant on the IC or
mod_interchange version.  Perhaps it's an issue with RPC mode and
mod_interchange?  Or just a new attack that is in the wild?

-Ron