__/ [www.1-script.com] on Thursday 03 November 2005 16:55 \__
> Roy Schestowitz wrote:
>
>
>
>> I have noticed that MSN and Yahoo index less than 5% of the pages in
>> my
>> site while Google is 'aware' of the existence of them all. I tried
>> to
>> think if MSN and Yahoo are hindered by JavaScript, but there
>> are
>> JavaScript-free routes to all pages.
>
>> Why is it that Yahoo and MSN will not crawl deep or at least index
>> pages
>> that are buried deeper in the site? Judging by the logs, they
>> consume
>> (spider) just about as much traffic as Google. When it comes to
>> referrals,
>> Google delivers almost 100 times the amount of referrals from Yahoo
>> and
>> MSN combined. How can this be?
>
>> Help appreciated,
>
>> Roy
>
> Looking at last 7 days data, I got 95% of SE traffic from Google, 2.2%
> from Yahoo and 0.6% from MSN.
Same here, roughly.
> Yet, the downloaded amount data goes like this:
> Yahoo! Slurp: 603.6Mb
> msnbot: 278.5MB
> Googlebot (Freshbot):112.7MB
> Googlebot (Deepbot): 155.9Mb
Similar figures to mine, but exchange Yahoo with Google. Google Images
account for much of the traffic from Google.
> The only way I can see this happening is that Y and MSN do not deem my
> pages important enough to include in the indices, but still important
> enough to keep banging on them. I would already disallow both Y! and MSN
> if I would not be #1 on one of the most important keywords in both. That
> said, however, I have to admit that being #1 on Y! or MSN only warms one's
> heart and nothing else :-) as far as traffic is concerned.
I have done some further work yesterday, trying to figure how this could
be attributed to.
Smaller sites of mine are fully indexed by all (almost identical numbers
across search engines). I looked up some other sites which contain many
pages. I could observe that it is not only me who is affected. Maybe it is
the innate behaviour of the algorithms? Imposing a limit on number of
pages? If so, it's an interesting situation.
Google focuses on breadth and will therefore be able to provide
answers/solutions to many answers/problems whereas Yahoo and MSN are nar-
row, focused, and more specific. They concentrate on fewer 'quality' pages
at the expense of sparseness of 'knowledge'.
Just try 'site:http://example.org' which works in MSN and Google. The dif-
ference is rather glaring to to the eye. In MSN and Yahoo, performing
searches that target deep pages will produce nothing, which implies that
the indices are indeed depleted. It is not the case of misleading page
count (saturation). Since Yahoo and MSN's combined share of the 'pie' is
approximately equal to that of Google, I was wishin' and hopin'...
Roy
|
|
