Roy Schestowitz wrote:
>> They also misbehave and do not honor Crawl-delay setting.
I have observed the opposite. They DO behave well in terms of robots.txt
directives handling. The key is: the robot.txt is cached, and I do not
think they update it as fast as they say they do. I would say the real
life cache span for Google is at least 48 hours or even more, so if you
added crawl-delay recently, it may not be in effect as yet.
Did you type it right?
User-Agent: Googlebot
Crawl-Delay: 30
> It is not quite possible to limit the demands that reach from Google if I
> recall correctly. That was the conclusion reached, I think.
You can' staisfy Google's thirst for knowledge (or world search engine
domination - pick one), but you can make them consume your bandwidth
slower. Like I said above, they do follow crawl-delay directive, but it
takes a long delay (no pun) to take the directive in.
--
Cheers,
Dmitri
See Site Sig Below
-------------------------------------
--
##-----------------------------------------------##
Article posted with Web Developer's USENET Archive
http://www.1-script.com/forums
Web and RSS gateway to your favorite newsgroup -
alt.internet.search-engines - 15980 messages and counting!
##-----------------------------------------------##
|
|
