__/ [ Phil Payne ] on Sunday 30 April 2006 11:34 \__
> __/ [ www.1-script.com ] on Friday 28 April 2006 21:08 \__
>
> > Roy Schestowitz wrote:
> >
> >> For the past week they have been crawling a lot. I haven't checked all
> >> my
> >> sites and I only noticed that last night. Earlier this month it was MSN
> >> that
> >> ate up over a gig in just a couple of weeks. Google is only third in
> >> terms
> >> of activity. Imagine that...
> >
> > Tell me about it! This month I fed Yahoo Slurp alone 16.58 GB! My entire
> > human traffic according to Urchin was 20.8 GB this month, and that
> > includes pictures that bots don't download. Total bot traffic, however
> > (only including top 10) was 25.19GB so far this month. It is clearly
> > becoming bots feeding frenzy!
>
> Worse here. The bots take over 75% of my bandwidth.
When you spread breadbrumbs (to analogize pages) all over your garden, the
crows will come. I think moderation in terms of scale is one way to prevent
crawlers from exploring. Another possibility is to indicate that pages are
static in the sense that they are no longer being extended. See:
http://www.i18nguy.com/markup/metatags.html
More crawling does not /necessarily/ mean more visitors. It's where crawling
efforts are concentrated (affect cycles/frequency of update) that matter.
With kind regards,
Roy
--
Roy S. Schestowitz | "Error, no keyboard - press F1 to continue"
http://Schestowitz.com | SuSE Linux ¦ PGP-Key: 0x74572E8E
11:40am up 2 days 18:37, 14 users, load average: 1.01, 1.09, 0.88
http://iuron.com - next generation of search paradigms
|
|
