[ic] RobotIPs

Jonathan Clark jonc at webmaint.com
Fri Feb 20 16:19:45 EST 2004


interchange-users-bounces at icdevgroup.org <> wrote:
> Jon [prtyof5 at attglobal.net] wrote:
>>>> RobotIP <<EOR
>>>>     202.9.155.123,      204.152.191.41,         208.146.26.19,
>>>>     208.146.26.233,     209.185.141.209,        209.185.141.211,
>>>>     209.202.148.36,     209.202.148.41,         216.200.130.207,
>>>>     216.35.103.6?,      216.35.103.70,
>>>> EOR
>>>>
>>>> Seems like there should be some 64.68*
>>>> in there shouldn't there ??
>>>>
>>> Please be so kind to tell us which robots crawling from this IP
>>> block to help the ICDEVGROUP to improve the default settings.
>>>
>> Googlebot.   Or am I wrong ?  Assuming so would it be correct
>> and appropriate to add    64.68?  ??
>>
> 64.68.* would cover 65,536 IP addresses.  It's probably safe to say
> that they're not all used by spiders.  They're definitely not all
> owned by Google. :-)
>
> For instance, 64.68.0.0/19 is allocated to by "Elan Communications".
> 64.68.32.0/19 is allocated to "Advanced Access" and 64.68.64.0/19
> is allocated to "Cable & Wireless".  Within the Cable & Wireless
> block, 64.68.78.0/23 is reallocated (sub-let) to eBay.  I believe
> Google has a large block in there somewhere (something like 2048 IPs)
> but I can't find it at the moment, and they're probably not all used
> by spiders anyway.
>
> A more focused list of addresses may make it into the default set,
> as long as a proper justification is presented.  You're free to add
> any addresses you like into your own local configuration, of course.

And this would help anyone wanting to do that:

http://www.searchengineworld.com/spiders/spider_ips.htm

Google does have more blocks listed, but the referrer id remains the same:

	Googlebot/2.1 (+http://www.googlebot.com/bot.html)

at the moment so IC should behave properly anyway. Out of interest, we are
being spidered from 64.68.82.0/24 at the moment. One server has this block
added, one doesn't. Both are behaving properly for google.

Jonathan.



--
Jonathan Clark
Webmaint.com - Building Clever Websites        http://www.webmaint.com/
Webmaint.net - Business Web Hosting            http://www.webmaint.net/
WireTel      - Internet Connectivity Solutions http://www.wiretel.net/




More information about the interchange-users mailing list