CarolW. wrote:
> On Wed, 12 Jan 2005 22:56:54 GMT, "star green" <spam@verizon.net>
> wrote:
>
>>Ive been trying to figure out how to write a robots.txt file that will
>>allow the robots to access the homepage (index.html), but no other page on
>>the site.
>>
>>Short of putting in a "disallow" line for every webpage on the site,
>>except for the homepage, is there some easy way to do this?
>
> One suggestion: Could put the other pages into a subdirectory folder
> then disallow the spiders from those folders.
>
> Carol
This can be a big overhaul in the absence of _relative_ links.
I used the following excellent source for all my robots.txt needs:
http://www.robotstxt.org/wc/faq.html#log
--
Roy Schestowitz
http://schestowitz.com
|
|
