__/ [ David Cary Hart ] on Friday 12 May 2006 17:01 \__
> Who is using the UA and why? I cannot even find were to obtain it.
> This seems to be a Hoover yet I have yet to see an instance where
> any have checked robots.txt. I have been redirecting these. Am I
> losing legitimate traffic?
What's the nature of the site? MATLAB, whose is heavily based on Java, can
be used as a (fairly rudimentary) Web browser, so denying it might not be
a good idea. It is primarily used for up-to-date documentation. That is
why I am asking about the nature of your site.
Additionally, see the if the paths (sequence of requested files) seems to
characterise these as human visitors rather than some experimental bot, a
lamer, or a leech. Don't neglect the possibility of spoofing. In the past
week alone, two people in the search engine newsgroups reported being
whacked by Google or Yahoo. Upon closer inspection, there were probably
fakers (at least one of them confirmed).
wget -R --user-agent="Java/1.2.4_55" your_site_URL
Simple, no? *smile*
Hope this helps,
Roy
--
Roy S. Schestowitz
http://Schestowitz.com | SuSE Linux ¦ PGP-Key: 0x74572E8E
5:05pm up 15 days 0:02, 9 users, load average: 1.18, 0.95, 0.76
http://iuron.com - Open Source knowledge engine project
|
|
