[ic] Spiders and more lists again
DB at M-and-D.com
Thu Sep 2 14:10:03 UTC 2010
This has been discussed before. Spiders crawling "more list" links
sometimes put a significant load on my server. I'm trying to identify
exactly which of my pages that the spiders are trying to crawl so that I
can modify the page(s), rewrite the URLs or otherwise fix the problem.
An example of such a request from my httpd access log is:
If I try this URL in a browser, I get "no search was found" Can anyone
provide a clue about what exactly the spider is looking for and/or come
up with a clever solution?
More information about the interchange-users