__/ [ www.1-script.com ] on Tuesday 02 May 2006 16:43 \__
> I hope this was not discussed to death here since it's so apparent, but
> it's news to me.
>
> Looking at Google's own robots.txt here (http://www.google.com/robots.txt)
>
>
> User-agent: *
> Allow: /searchhistory/
> Disallow: /search
> Disallow: /groups
> Disallow: /images
>
> and the list goes on...
>
>
> What? Allow: ? Since when there is an Allow directive in the set?
It's in the specifications as far as I know. It always has been (at least
ever since I become familiar with robots.txt). Maybe it is used as an
indication/emphasis of availability or somewhat of a directive like XML
sitemaps.
Here is one relevant reference:
http://www.freefind.com/library/howto/robots/
,----[ Quote ]
| It is also possible to use an "allow" in addition to disallows.
| For example:
|
| user-agent: FreeFind
| disallow: /cgi-bin/
| allow: /cgi-bin/Ultimate.cgi
| allow: /cgi-bin/forumdisplay.cgi
`----
The example above comes to show you when "allow" can be useful, under
particular circumstances. "None apart from" or "All except for".
Best wishes,
Roy
--
Roy S. Schestowitz | SuSE, Mandriva, Fedora - Gotta love them girls
http://Schestowitz.com | SuSE Linux ¦ PGP-Key: 0x74572E8E
5:50pm up 5 days 0:47, 11 users, load average: 0.23, 0.46, 0.49
http://iuron.com - Open Source knowledge engine project
|
|
