[ic] Re: interchange-users Digest, Vol 47, Issue 18

[Carl Bailey]

On May 24, 2007, at 12:00 PM, kevin at cursor.biz wrote:

> Date: Wed, 23 May 2007 19:24:40 +0100
> From: Kevin Walsh <kevin at cursor.biz>
> Subject: Re: [ic] Rolling big tables (mysql)
> To: interchange-users at icdevgroup.org
> Message-ID: <200705231924.41590.kevin at cursor.biz>
> Content-Type: text/plain;  charset="iso-8859-1"
>
> Grant <emailgrant at gmail.com> wrote:
>>>> I do keep a separate table of robot UAs and match traffic rows to  
>>>> them
>>>> with op=eq to populate another table with robot IPs and non-robot  
>>>> IPs
>>>> for the day to speed up the report.  Don't you think it would be
>>>> slower to match/no-match each IC request to a known robot UA and  
>>>> write
>>>> to the traffic table based on that, instead of unconditionally  
>>>> writing
>>>> all requests to the traffic table?  If not, excluding the robot
>>>> requests from the traffic table would mean a lot less processing for
>>>> the report and a lot fewer records for the traffic table.
>>>>
>>> Perhaps you should create a column called "spider" in the traffic  
>>> table
>>> and save a true or false value depending upon the [data session  
>>> spider]
>>> value.  You can then generate reports "WHERE spider = 0", for  
>>> ordinary
>>> users, or "WHERE spider = 1" for robots etc.  An index on the spider  
>>> column
>>> would be nice, of course.
>>
>> I let this roll around in my head for quite a while and I ended up
>> writing the IC page accesses to my traffic table based on [data
>> session spider] like you suggested.  This should mean a much smaller
>> traffic table and less processing when running a report on it.  We'll
>> see how much time it buys me before running the report takes too long
>> again.  I also need to set up indexes.
>>
> Also, you may as well grab the latest robots.cfg file from CVS and
> "include" it into your interchange.cfg file.

... and where would that be in CVS?  I read the MANIFEST and it cited  
only /debian/robot.cfg but the file is not present here:
http://www.icdevgroup.org/cgi-bin/cvsweb/interchange/debian/? 
only_with_tag=REL_5_4_2

I guess either the file is missing or the manifest is wrong.

Grateful if you can point me in the right direction.

Carl
- - - - - - - - - - - - -
  Carl Bailey
  Triangle Research, Inc.
  tel: 919.323.8025
- - - - - - - - - - - - -