star green wrote:
> "John Bokma" <postmaster@castleamber.com> wrote in message
> Xns95DE3D1AE72castleamber@130.133.1.4">news:Xns95DE3D1AE72castleamber@130.133.1.4...
>> Roy Schestowitz wrote:
>>
>>> star green wrote:
>>>
>>>> "John Bokma" <postmaster@castleamber.com> wrote in message
>>>> news:Xns95DDD881F421castleamber@130.133.1.4...
>>>>> star green wrote:
>>>>>
>>>>>> Ive been trying to figure out how to write a robots.txt file that
>>>>>> will allow the robots to access the homepage (index.html), but no
>>>>>> other page on the site.
>>>>>
>>>>> Note that not all robots honor robots.txt. The major search engine
>>>>> ones do (most likely), but ratware bots don't.
>>>>>
>>>>> --
>>>>> John
>>>>
>>>> I know, I was just wondering if there's something that would work for
>>>> most robots.
>>>>
>>>> But, I guess there's nothing to be done short of completely
>>>> reorganizing the site (which I don't wish to do) or listing every
>>>> page in the robots.txt file.
>>>> Ah, well.
>>>>
>>>> Thanks to everyone for the advice!
>>>
>>> Use a script. You can get a listing of all HTML files on your site and
>>> output them to a file. Contact me if you need help on this.
>>
>> Script?
>>
>> find . -name "*.html" -print and pipe that into sed :-D
>>
>>> Having said that, if your site has, let us say, 10,000 pages, then
>>> robots.txt becomes overly inflated.
>>
>> You can probably mod_rewrite to a cgi script that generates robots.txt
>> on the fly :-D
>>
>> --
>> John ->
>
> Roy & John,
>
> thanks for your suggestions about a script - but I haven't the foggiest
> idea how I'd go about creating, finding, or implementing such a thing.
> ;--)
>
> I think I'll just go ahead and make that robots.txt that excludes all but
> the one webpage I want to be found ~ appreciate everyone's help.
>
> best,
> star.
Look at my recent reply that's assigned to the OP's first message. Some
simple idea that came up while I was in the sauna...
--
Roy Schestowitz
http://schestowitz.com
|
|
