[ic] mod_interchange and Apache MaxClients

[John1]

On Wednesday, November 16, 2005 6:00 PM, rphipps at reliant-solutions.com 
wrote:

>> From: interchange-users-bounces at icdevgroup.org
>> [mailto:interchange-users- bounces at icdevgroup.org] On Behalf Of John1
>> Sent: Wednesday, November 16, 2005 8:05 AM
>>
>> On Wednesday, November 16, 2005 12:32 AM, list_subscriber at yahoo.co.uk
>> wrote:
>>
>>>> Last night I recompiled with MaxClients to 1024 and this morning I
>>>> ran into the problem again, also MaxClients was not reached this
>>>> time, however I was unable to reach the IC site.  At least in my
>>>> case I do not believe MaxClients to be the issue.
>>>>
>>> I now agree - after some investigation I think that MaxClients issue
>>> is a symptom rather than the cause.  No doubt if you have increased
>>> MaxClients to 1024 you are now not hitting this limit, but are upi
>>> still seeing a high Apache client count??  I think that once the
>>> Interchange daemon
>>> starts to have problems this must have a cascade effect on the
>>> Apache processes
>>> being spawned
>>>
>> I believe that my site only actually ceases to respond once
>> MaxClients is reached.  Even though MaxClients is no doubt not the
>> underlying cause of the
>> problem (but a symptom) what I would like to do is automatically
>> restart apache and interchange when MaxClients is reached as this
>> would keep website
>> downtime to a minimum.
>>
>> It occurs to me that a good way of doing this would be to pipe the
>> Apache error log through a script which greps for "server reached
>> MaxClients setting, consider raising the MaxClients setting".
>>
>> As soon as the script sees the above error message being piped to it
>> I would
>> like it to restart the services without delay.
>>
>> Is this possible?  I was wondering if anyone knows how to do this?
>> e.g. an
>> example script.  If so, I would be very grateful as I am currently
>> clueless
>> on how to attempt this.  Thank you.
>
> John,
>
> I wrote a script yesterday that runs via cron and what it does is it
> checks if a certain ic page returns a certain string.  If the page
> does not return properly it tries again, up to 5 times.  If after 5
> times it doesn't get the page back it restarts Interchange as well as
> emails an address to alert that the site restarted.  This way we see
> the problem before MaxClients is reached since I have figured out
> that MaxClients is not reached until some undetermined amount of time
> after the site stops processing requests.
>
So have you found/confirmed that the client count does keeping ticking 
upwards once the site stops processing requests?

> This morning the site went
> down and my script immediately noticed the problem and restarted IC
> properly.  Of course this is just to keep the site up, we still need
> to figure out why it's happening.
>
> Below is the script:
>
> #!/usr/bin/perl
>
> use LWP;
>
> my $url = 'http://www.yoursite.com/checkic.html';
>
> ## CMD to restart
> my $IC_COMMAND = "/root/scripts/icrestart.sh";
>
> ## Where is sendmail?
> my $SENDMAIL_CMD = "/usr/sbin/sendmail";
>
> ## what mail account should we alert to?
> my $MAIL_RECEIVER = "email\@yoursite.com";
>
> ## what should the subject be?
> my $MAIL_SUBJECT="Alert!";
>
> ## who should we send from?
> my $MAIL_SENDER="root\@yoursite.com";
>
> my $browser = LWP::UserAgent->new;
> $browser->timeout(30);
>
> my $count = 0;
> my $up = 0;
>
> while ($count <= 4) {
>        my $response = $browser->get($url);
>        if ($response->content =~ m/UP/) {
>                $count = 5;
>                $up = 1;
>        }
>        $count++;
> }
>
> if ($up == 1) {
> } else {
>        system $IC_COMMAND;
>        mail_admin("IC restarted!");
>        print "IC restarted!";
> }
>
> sub mail_admin {
>        my $mail = "From: $MAIL_SENDER <>\nTo: $MAIL_RECEIVER\nSubject:
> $MAIL_SUBJECT\nX-Priority: 1\n\n$_[0]\n";
>        print STDERR $mail if $DEBUG;
>        return if $NO_MAIL;
>        open SENDMAIL, "| $SENDMAIL_CMD -t" and
>        print SENDMAIL $mail and
>        close SENDMAIL  or
>        die "Failed to send alert mail to $MAIL_SENDER: $!";
> }
>
>
> Then put a page "checkic.html" on your site with the content of "UP"
> in the pages directory and put the url to this page in the $url
> variable above.
>
Fantastic, thanks Ron, just implemented it!  It occurs to me that it may be 
useful to call a few system commands before restarting interchange, and to 
add the output from these commands to the alert e-mail sent.

e.g.

#Number of connections to Apache just before Interchange is restarted
netstat -nt | grep :80 | wc -l
netstat -nt | grep :443 | wc -l

#Number of httpd, interchange and mysql processes running just before 
Interchange is restarted
ps -elf | grep -c httpd
ps -elf | grep -c interchange
ps -elf | grep -c mysqld

#Number of connections each IP address has to server just before Interchange 
is restarted
netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

Although I could just about work out how to insert these into your script it 
would involve educated guesswork and no doubt ugly coding, so would be 
grateful if you could insert these system calls or any other similar or 
additional commands that you think may provide useful information.

> You could modify the checkic.html page to also do db lookups to verify
> the db connection is still valid.
>
If you think this would be useful would you mind also adding this as my Perl 
isn't up to it.

Thanks for you help!


		
___________________________________________________________ 
To help you stay safe and secure online, we've developed the all new Yahoo! Security Centre. http://uk.security.yahoo.com