__/ [ Borek ] on Monday 15 May 2006 10:20 \__
> On Mon, 15 May 2006 11:12:28 +0200, Phil Payne <phil@xxxxxxxxxxxxxxxxxxxx>
> wrote:
>
>> Going through this month's log I've found lots of search engine bot
>> visits that have downloaded robots.txt and index.html - and nothing
>> else.
>
> If you will report it for Google only it will be not surprising, as that's
> kind of repoert we are bombarded on hourly basis lately on aise. But if
> all bots behave this way - are you sure you are not blocking access with
> robots.txt, or with 404 or something?
As it's being download and then index.html get fetched, this seems unlikely.
News sites tend to observe this behaviour, initially at least. Are you by
any chance using JavaScript navigation?
Best wishes,
Roy
--
Roy S. Schestowitz | "Computers are useless. They only solve problems"
http://Schestowitz.com | GNU is Not UNIX ¦ PGP-Key: 0x74572E8E
10:50am up 17 days 17:47, 8 users, load average: 0.87, 0.61, 0.48
http://iuron.com - proposing a non-profit search engine
|
|
