Monday, June 27th, 2005, 5:00 pm
Googlebot versus MSNBot
Filed under:
Bill Gates arrested in his younger days (photo in public domain)
n item I have just come across (hat tip Justin Moore) speaks about excluding MSNBot based on poor performance. MSNBot (and Yahoo! likewise) consume a great deal of bandwidth, but are barely used as search engines by the Internet audience.
From Justin:
- Exhibit A – MSNbot Crawl hits = 9561
- Bandwidth used = 124.43 MB
- Visits to wantmoore.com as a results from searches @ Google = 683* [sic.*]
- Exhibit B – Googlebot Crawl hits = 3415
- Bandwidth used = 51.74 MB
- Visits to wantmoore.com as a results from searches @ MSN = 41 [sic.*]
…
So, effective immediately:
User-agent: MSNBot
Disallow: /* Order of the visits is wrong and should be reversed.
I spotted similar statistics on my domain, with Google as the top referrer at almost 20,000 visits/month. MSN remain at a miserable 200 and take almost as much bandwidth as Googlebot. I have plenty of bandwidth to spare; So, I might as well let Microsoft spend crawling time in vain.
June 28th, 2005 at 5:54 am
Wait, wait, wait, you’re blocking MSN bot from using a whole 100mb of bandwidth, because it only sends you 41 hits / month? Who cares, I mean it’s 100mb of bandwidth, just let it send you that wee bit of traffic? or, of course you could SEO for MSN…
June 28th, 2005 at 7:46 am
Googlebot versus MSNBot
[Source: schestowitz.com] quoted: Wait, wait, wait, you’re blocking MSN bot from using a whole 100mb of bandwidth, because it only sends you 41 hits / month? Who cares, I mean it’s 100mb of bandwidth, just let it send you that wee bit of tr…
June 28th, 2005 at 8:51 am
Webmaster (Floobo),
No, no, no, it was Justin Moore whose choice was to do that (it was a blockquote, in fact). I very much disagree to putting robots in a position of disadvantage. I don’t discriminate against crawlers even if their potential is low.
Allowing no access to competition is the same evil that resulted in proprietary protocols, from which we all suffer.