>> By that logic, should robots like AdsBot and SurveyBot be added to >> robots.cfg? > > > If you've confirmed they're real bots visiting your site, yes, I think so. > > > Jon Ok, I understand the concept then. I'll put together some changes into a patch and post here. - Grant