[ic] New Robot UAs

Grant emailgrant at gmail.com
Sat Dec 31 00:11:08 UTC 2011


>>>> I've been going through Apache data lately and have discovered at least
>>>> two new robots that ought to be added to robots.cfg:
>>>>
>>>> 1) 'bingbot'
>>>>
>>>>
>>>> http://www.bing.com/toolbox/blogs/webmaster/archive/2010/06/28/bing-crawler-bingbot-on-the-horizon.aspx
>>>>
>>>> Looks like Microsoft has changed the UA for MSNBot, but Interchange is
>>>> no longer recognizing it.
>>>
>>>
>>> It should still recognize it, as the simple term "bot" is in the RobotUA
>>> list. Neverless I added it.
>>
>>
>> If bingbot in robots.cfg doesn't change any behavior, I suggest it is
>> removed.
>
>
> I disagree: I think it's better to explicitly add bingbot even if bot is
> already there, because later someone thinking "bot" is overly broad would be
> likely to remove it with no idea that they're now making "bingbot" fail to
> be recognized as a robot.
>
> Jon

By that logic, should robots like AdsBot and SurveyBot be added to robots.cfg?

- Grant



More information about the interchange-users mailing list