[ic] Where is robots.cfg included?
Peter
peter at pajamian.dhs.org
Wed Mar 3 01:32:49 UTC 2010
On 03/03/10 13:10, Grant wrote:
>>> I'd like to switch from my manually-built RobotsUA definition to the
>>> IC robots.cfg file. I thought it would be included from catalog.cfg,
>>> but I don't see a reference to it in the latest version. Where is
>>> robots.cfg called from?
>>>
>> It's global, see:
>>
>> http://github.com/interchange/interchange/blob/master/dist/interchange.cfg.d
>> ist
>>
>> # Include individual usertags.
>> include usertag/*.tag
>>
>> # Include the robot recognition lists
>> include robots.cfg
>>
>> # Include the list of country-specific standardised subdomains
>> include subdomains.cfg
>
> Thank you Gert. Has anyone run into problems relying on robots.cfg?
I've never had problems. Others have occasionally had problems with
things such as recently the ask toolbar in a browser being confused for
the ask.com spider bot (note this has been fixed in robots.cfg now).
Peter
More information about the interchange-users
mailing list