[ic] New Robot UAs
Jon Jensen
jon at endpoint.com
Fri Dec 30 01:12:06 UTC 2011
On Thu, 29 Dec 2011, Grant wrote:
>>> I've been going through Apache data lately and have discovered at
>>> least two new robots that ought to be added to robots.cfg:
>>>
>>> 1) 'bingbot'
>>>
>>> http://www.bing.com/toolbox/blogs/webmaster/archive/2010/06/28/bing-crawler-bingbot-on-the-horizon.aspx
>>>
>>> Looks like Microsoft has changed the UA for MSNBot, but Interchange is
>>> no longer recognizing it.
>>
>> It should still recognize it, as the simple term "bot" is in the
>> RobotUA list. Neverless I added it.
>
> If bingbot in robots.cfg doesn't change any behavior, I suggest it is
> removed.
I disagree: I think it's better to explicitly add bingbot even if bot is
already there, because later someone thinking "bot" is overly broad would
be likely to remove it with no idea that they're now making "bingbot" fail
to be recognized as a robot.
Jon
--
Jon Jensen
End Point Corporation
http://www.endpoint.com/
More information about the interchange-users
mailing list