John Bokma wrote:
> mogga <di@NOSPAMPLEASEmogga.com> wrote:
>
>> On Wed, 20 Jul 2005 11:09:17 GMT, info_at_1-script_dot_com@foo.com
>> (www.1-script.com) wrote:
>
> [ .. ]
>
>>>What's INSIDE that robot.txt file?
>>
>> Nothing.
>> I've checked too. Just in case.
>
> Copy the robots.txt of http://johnbokma.com/ . Although I doubt that an
> empty file hurts Googlebot.
The rules, which I had a long glance at in the past, state that it should
make no difference. The search engines are free to treat robots.txt any way
they like though.
I added a small exclusion to my site a month ago (mentioning names of senior
staff I work with) and this does not appear to have affected traffic from
Google. One would think that the actual 'invitation' for crawlers should
increase traffic, but I don't think it does. Not based on what I have
observed... empty robots.txt file, which does not exist on new sites by
default, should be a sufficient invitation.
Roy
--
Roy S. Schestowitz
http://Schestowitz.com
|
|
