[ic] Time for new hardware?

[Grant]

>> >>>>>> I've been making a lot of optimizations lately and I think I'm to the
>> >>>>>> point where my 4 cores just aren't able to keep up with demand during
>> >>>>>> peak traffic hours each day.  Pages load quickly when I see fewer than
>> >>>>>> 4 busy interchange processes in top but things slow down drastically
>> >>>>>> after that.  Once I'm OK with the degree of optimization my ITL pages
>> >>>>>> have undergone and I'm not IO-bound or memory-bound, is it time to
>> >>>>>> throw CPU at the problem?
>> >>>>>>
>> >>>>>> I'm a little puzzled because I've seen my server perform much better
>> >>>>>> under much greater loads in the past.  I thought my growing mysql
>> >>>>>> tables could be the problem so I set up indexes and it has helped but
>> >>>>>> my server still struggles under loads it used to handle without issue.
>> >>>>>> I did notice that my tables are split about 50/50 between InnoDB and
>> >>>>>> MyISAM and I'd like to make all of them InnoDB.
>> >>>>>>
>> >>>>>> - Grant
>> >>>>>
>> >>>>> Yeah, I would recommend to switch all them to InnoDB for consistency
>> >>>>> and real transactions.
>> >>>>>
>> >>>>> You probably can find sufficient resources on MySQL optimization with
>> >>>>> InnoDB if that is your bottleneck.
>> >>>>
>> >>>>
>> >>>> Actually IO doesn't seem to be a bottleneck at this point.  It seems
>> >>>> to be CPU as things slow down once I have 4 busy interchange processes
>> >>>> on my quad-core CPU and iotop does not show much activity at that
>> >>>> point.  Besides ITL optimization, is this a clear case of needing a
>> >>>> faster CPU and more cores or is there anything else to consider first?
>> >>>>
>> >>>>
>> >>>>> May I ask which webserver you are using?
>> >>>>
>> >>>>
>> >>>> I'm using nginx reverse proxied to apache2.
>> >>>
>> >>> Did you consider to get rid of Apache?
>> >>
>> >>
>> >> I'm certainly planning to do that but if I watch top it looks like my
>> >> apache2 processes *barely* get above idle even when the web server is
>> >> under heavy load.  The interchange processes get hammered.
>> >>
>> >
>> > Which mode/settings do you use for the Interchange server? How
>> > do you connect Nginx and Interchange?
>>
>>
>> Here are my active traffic settings from interchange.cfg:
>>
>> PreFork                 Yes
>> StartServers            12
>> MaxServers              0
>> MaxRequestsPerChild     100
>> HouseKeeping            2
>> PIDcheck                3600
>> ChildLife               3600
>>
>> apache2 listens on a particular port and nginx sends page requests to
>> apache2 over that port via the proxy_pass directive.  apache2 then
>> hands off to interchange via Interchange::Link.
>
> Apache is almost certainly not your problem.
>
> CPU usage is not always Interchange -- your database uses CPU.


Could the database be to blame if the interchange processes are the
ones using all the CPU in top?  Would top show mysql using CPU if the
database were to blame?


> My top suspect is SQL queries that are not optimized. In my experience,
> adding a few indices to help key queries usually solves this type of
> problem.


I added a lot of table indexes lately and it did speed things up a lot
but it didn't solve the problem of http response time slowdowns during
peak traffic hours.

- Grant