__/ [ Phil Payne ] on Friday 05 May 2006 11:17 \__
> At around 09:00 UTC this morning.
>
> First it tried to get 0T717Q3K81F45P9CHK78.htm - obviously a 404
> functionality test.
>
> Then it proceeded to download the site. Yes, all of it. We'll see
> what turns up in the SERPs.
How many pages in total? Googlebot never appears to do 404 tests. Neither
do MSNBot, Yahoo/Inktom Slurp and other noticeable spiders (albeit Yahoo
used to be so buggy, so it crawled incorrectly to request wrong files from
the wrong sites). What I am trying to suggest that somebody may have
forged user-agent. It's very simple to do this. It gives a cloak of
stealth to someone wishing to rip off your site entirely, possibly using a
grabber, e.g.
wget -R --user-agent="Googlebot whatever..." your_site_URL
Best wishes,
Roy
--
Roy S. Schestowitz, Ph.D. Candidate (Medical Biophysics)
http://Schestowitz.com | Free as in Free Beer ¦ PGP-Key: 0x74572E8E
11:15am up 7 days 18:12, 12 users, load average: 1.02, 0.86, 0.76
http://iuron.com - semantic engine to gather information
|
|
