[ic] add to robots.cfg ?

Peter peter at pajamian.dhs.org
Thu Dec 19 01:05:56 UTC 2013

On 12/19/2013 11:08 AM, Justin La Sotten wrote:
>> www.domain.com - [18/Dec/2013:14:42:01 -0500] "GET
>> /page.html HTTP/1.1" 200 26070 "-" "msnbot-UDiscovery/2.0b
>> (+http://search.msn.com/msnbot.htm)"
> Hmm, that's odd, did MSN change their bot UA back again?
> It used to be called "msnbot" but then they changed it to "bingbot".
> And now it's back to "msnbot" ... odd.

It wouldn't surprise me if there are still some servers that were simply
never changed in the first place.


More information about the interchange-users mailing list