[ic] Interchange using lots of cpu...

[interchange-users@icdevgroup.org]

Mike Heins <mike@perusion.com> replied:
> 
>     http://www.icdevgroup.org/i/dev/docfly.html?mv_arg=icfaq14.02

Thanks -- I'd already found that page which gave valuable pointers on
what to do to avoid overall efficiency problems.  I'm using its
[benchmark] code to get my benchmarks which was very helpful.  I
didn't want to reach its final stage of simply using [query] to
get the results, then writing all the list output in perl.  I
might have to do that though. :(

> Besides the FAQ I posted, you are calling at least three Perl interpreters
> per result (the [if scratch foo eq bar] stuff). With lots of results, this
> will result in unacceptable processor use.
> 
> Also, since you haven't mentioned it, one wonders at the status of the
> indexes in your database.

All good points, but the main problem here is that the loop runs
acceptably quickly *per*iteration* with small result sets.  When the
result set gets bigger the loop runs unacceptably slowly *per*iteration*
initially and gradually speeds up to its previous speed as results are output.

It sounds like something somewhere is copying either the result set or
something associated with it with some nasty quadratic behavior.  It
wouldn't be Safe.pm would it?  Or some other consequence of firing
up perl?  I feel that this is the most likely explanation at this point.

And, as it's interchange that's consuming cpu, I can't see a problem
at the mysql or database end.  The database isn't even very big!  Doing
the select by hand in mysql produces all the 500 lines of output in 0.02
seconds elapsed.  None of the [ARTICLE-param x] calls are causing
interchange to perform more db lookups are they?  They're just
using the already-returned result set directly?

- gordon