Home Messages Index
[Date Prev][Date Next][Thread Prev][Thread Next]
Author IndexDate IndexThread Index

Re: Deleted a Site and Crawlers Went Nuts?

John Bokma wrote:

> Roy Schestowitz wrote:
> 
>> John Bokma wrote:
>> 
>>> They probably retried because of the 404 they got.
>> 
>> That's what I thought but look at the bandwidth...
> 
> Day     Hits    Files   Pages   Visits  Sites   KBytes
> 22      2007    1816    1805    755     327     23846
> 
> Let's just assume that every hit resulted in a 404 page:
> 
> 23846 KByte / 2007 approx 12 kbyte/hit
> 
> A 404 page I requested from my site is just half a KB
> 
> Do you have a custom 404?

Good thinking, John. I have no custom 404 on that site. It's possible that
the bot went on to indexing the stats pages...

I am relieved now, so thanks.

Roy

-- 
Roy Schestowitz
http://schestowitz.com

[Date Prev][Date Next][Thread Prev][Thread Next]
Author IndexDate IndexThread Index