__/ [ www.1-script.com ] on Friday 28 April 2006 01:06 \__
> Is it just my turn today or you guys also see an enormous increase in
> Yahoo's crawling?
For the past week they have been crawling a lot. I haven't checked all my
sites and I only noticed that last night. Earlier this month it was MSN that
ate up over a gig in just a couple of weeks. Google is only third in terms
of activity. Imagine that...
> I would not be surprised if they will end up requesting 100,000 pages
> before the day's end (on a 500,000+ pages site). What are they, on a
> mission to re-crawl it all in less than a week or something?
You have over a quarter of a million pages on your site, according to what I
can see. Could you slow down the delivery pages selectively (i.e.
deliberately slowing it down for Yahoo)? The quicker you serve pages, the
better capacity is assumed and the more greedy crawlers can become, I think.
> It could also be that they are trying to beef up their index in order to
> beat Google in their "mine's (index) bigger" race.
One thing I noticed last week is that the number of pages Yahoo retain
(saturation) has increased significantly. The number of referrals increased
as well. Perhaps they go for depth and breath rather than highly-regarded
pages.
> What do you guys see in your stats?
Too much stuff for conclusions to be reached within a decent amount of time.
Measuremap, which is a stats tool for blogs (recently bought by Google),
attempts to summarise detailed statistics and translate them into simpler,
digestable, informative lumps. Maybe it's a smart way to go. Imagine a stats
package that indicates (using flag) where there are changes in general
changes in behaviour (trends). I don't know Urhcin beyond its demos...
Hope it helps,
Roy
--
Roy S. Schestowitz
http://Schestowitz.com | SuSE Linux ¦ PGP-Key: 0x74572E8E
8:40am up 15:45, 12 users, load average: 0.42, 0.37, 0.41
http://iuron.com - next generation of search paradigms
|
|
