Paul Burke wrote:
> On Fri, 15 Jul 2005 20:41:44 +0100, mogga <di@NOSPAMPLEASEmogga.com>
> wrote:
>
>>Robots/Spiders visitors (Top 25)
>>
>>In my stats (awstats)
>>I have a section about robots and spiders.
>>
>>* Robots shown here gave hits or traffic "not viewed" by visitors, so
>>they are not included in other charts. Numbers after + are successful
>>hits on "robots.txt" files
>>
>>Why does it count the successful hits on robots.txt and what purpose
>>does it have?
>
> It could be human. I always look for a robots.txt when someone asks
> for a link exchange, or me checking a site out to exchange links with.
> hth
> plh
> Paul
I have had various user agents access non-existent URL's like robot.txt or
robots1.txt. It usually raises questions and leads to a short investigation
or research on the Web, which is a total waste of time. There are also some
experimental bots that make nothing but a mess in the error logs. Always
bear that in mind if you see suspicious stuff going on. It is nothing to
worry about.
Roy
--
Roy S. Schestowitz
http://Schestowitz.com
|
|
