[ic] New Robot UAs

Jon Jensen jon at endpoint.com
Fri Dec 30 01:12:06 UTC 2011

On Thu, 29 Dec 2011, Grant wrote:

>>> I've been going through Apache data lately and have discovered at 
>>> least two new robots that ought to be added to robots.cfg:
>>> 1) 'bingbot'
>>> http://www.bing.com/toolbox/blogs/webmaster/archive/2010/06/28/bing-crawler-bingbot-on-the-horizon.aspx
>>> Looks like Microsoft has changed the UA for MSNBot, but Interchange is 
>>> no longer recognizing it.
>> It should still recognize it, as the simple term "bot" is in the 
>> RobotUA list. Neverless I added it.
> If bingbot in robots.cfg doesn't change any behavior, I suggest it is 
> removed.

I disagree: I think it's better to explicitly add bingbot even if bot is 
already there, because later someone thinking "bot" is overly broad would 
be likely to remove it with no idea that they're now making "bingbot" fail 
to be recognized as a robot.


Jon Jensen
End Point Corporation

More information about the interchange-users mailing list