Home Messages Index
[Date Prev][Date Next][Thread Prev][Thread Next]
Author IndexDate IndexThread Index

Re: robots.txt

  • Subject: Re: robots.txt
  • From: John Bokma <postmaster@castleamber.com>
  • Date: 14 Jan 2005 06:01:24 GMT
  • Newsgroups: alt.internet.search-engines
  • Organization: Castle Amber - software development
  • References: <WWhFd.6854$C.5005@trnddc05> <Xns95DDD881F421castleamber@> <xPBFd.15599$ig7.7937@trnddc04> <cs7f0c$1gc1$3@godfrey.mcc.ac.uk>
  • User-agent: Xnews/5.04.25
  • Xref: news.mcc.ac.uk alt.internet.search-engines:53818
Roy Schestowitz wrote:

> star green wrote:
>> "John Bokma" <postmaster@castleamber.com> wrote in message
>> news:Xns95DDD881F421castleamber@
>>> star green wrote:
>>>> Ive been trying to figure out how to write a robots.txt file that
>>>> will allow the robots to access the homepage (index.html), but no
>>>> other page on the site.
>>> Note that not all robots honor robots.txt. The major search engine
>>> ones do (most likely), but ratware bots don't.
>>> --
>>> John
>> I know, I was just wondering if there's something that would work for
>> most robots.
>> But, I guess there's nothing to be done short of completely
>> reorganizing the site (which I don't wish to do) or listing every
>> page in the robots.txt file.
>> Ah, well.
>> Thanks to everyone for the advice!
> Use a script. You can get a listing of all HTML files on your site and
> output them to a file. Contact me if you need help on this.


find . -name "*.html" -print and pipe that into sed :-D

> Having said that, if your site has, let us say, 10,000 pages, then
> robots.txt becomes overly inflated.

You can probably mod_rewrite to a cgi script that generates robots.txt 
on the fly :-D

John -> http://johnbokma.com/  Firefox: http://johnbokma.com/firefox/
                           Perl SEO tools: http://johnbokma.com/perl/
                 Experienced (web) developer: http://castleamber.com/
Get SEO help: http://johnbokma.com/websitedesign/seo-expert-help.html            

  • References:
[Date Prev][Date Next][Thread Prev][Thread Next]
Author IndexDate IndexThread Index