[ic] New Robot UAs

[Grant]

> Everyone,
>
> I've been going through Apache data lately and have discovered at
> least two new robots
> that ought to be added to robots.cfg:
>
> 1) 'bingbot'
> http://www.bing.com/toolbox/blogs/webmaster/archive/2010/06/28/bing-crawler-bingbot-on-the-horizon.aspx
>
> Looks like Microsoft has changed the UA for MSNBot, but Interchange is
> no longer recognizing it.
>
> 2) 'facebookexternalhit'
> http://www.facebook.com/externalhit_uatext.php

facebookexternalhit should be added to robots.cfg.  Here is Facebook's
explanation of how it's used:

"Facebook allows its users to send links to interesting web content to
other Facebook users. Part of how this works on the Facebook system
involves the temporary display of certain images or details related to
the web content, such as the title of the webpage or the embed tag of
a video. Our system retrieves this information only after a user
provides us with a link. You may have found this page because a
Facebook user sent a link from your website to other Facebook users.
If you have any questions or concerns about any links or content sent
by one of our users, please contact us at legal at facebook.com."

http://www.facebook.com/externalhit_uatext.php

Facebook itself retrieves the external page or image with the
facebookexternalhit UA so that UA shouldn't be given a session.
Should I submit a bug for this?

- Grant


> This is really only relevant for webmasters that have done some sort
> of facebook integration; ie. Like Button, Facebook Connect
>
>
> We'll continue pouring over the robot data, we'll let you know if we
> come up with anymore.
>
> --
> Regards,
> Justin La Sotten
> FragranceNet.com