[ic] optimizing ic cron jobs with lots of records to process

[Aaron]

Hi all,

IC 5.0

I have some cron jobs that run which create invoicing for a company.  It
works great, but the processing is taking too long and it is timing out
and killing ic.  It seems to me like it shouldn't be too many, but this
particular job could be going through a couple thousand records and
doing the processing on each one.

I have totally redone parts of this and put it all in perl which has
given me an improvement, but parts of it are not.  I will give you a
very general breakdown of what it is doing in the job.  I can make it
all work, but I need some input on what kind of things that I am doing
which are bad and causing the problems.

[query of companies to do invoicing for]
	[tmp test][sql-param bill_date][/tmp]
	[if type="scratch" term="test" op="==" compare="[scratch
today]"]
		[query of 1000 to 2000 records]
			[tmp][insert query][/tmp]
			[tmp][update query][/tmp]
			[if scratch..]
				[set var][sql-param..][/set]
			[else print for the cron log to have
information]

[if scratch var]
	<this part creates emails with the information that was
previously created above and I have redone this section to be just one
[perl][/per] tag and it is greatly improved.  It is basically just a sql
query, a foreach loop and it creates an email that will contain the one
to two thousand records.  There will only be one or two emails usually.
The problem is above because all of the records that SHOULD get
processed do not and it doesn't even get to the email portion before it
times out.>

You may want to see more, but I am just trying to keep it as simple as
possible.  Like I said, I need some direction on how to handle these
large lists that come from another list more efficiently so that I cut
down on the processing times.  This is just starting to be a problem,
but I need to correct because I want to be able to handle several
thousand with this same script in a short time.

One other thought, how much improvement would I see with parts of this
being done as UserTags.  Which portion should be set up that way?  Just
the inside processing with the update/insert?

Thanks,

--
Aaron