Googlebot versus MSNBot

Bill Gates arrested in his younger days (photo in public domain)
n item I have just come across (hat tip Justin Moore) speaks about excluding MSNBot based on poor performance. MSNBot (and Yahoo! likewise) consume a great deal of bandwidth, but are barely used as search engines by the Internet audience.
From Justin:
- Exhibit A – MSNbot Crawl hits = 9561
- Bandwidth used = 124.43 MB
- Visits to wantmoore.com as a results from searches @ Google = 683* [sic.*]
- Exhibit B – Googlebot Crawl hits = 3415
- Bandwidth used = 51.74 MB
- Visits to wantmoore.com as a results from searches @ MSN = 41 [sic.*]
…
So, effective immediately:
User-agent: MSNBot
Disallow: /* Order of the visits is wrong and should be reversed.
I spotted similar statistics on my domain, with Google as the top referrer at almost 20,000 visits/month. MSN remain at a miserable 200 and take almost as much bandwidth as Googlebot. I have plenty of bandwidth to spare; So, I might as well let Microsoft spend crawling time in vain.






Filed under: 

ry comparing news feeds from CNN against those from the BBC. I am subscribed to various feeds from both and in CNN no descriptions are bound to the titles. Why even syndicate if the subscriber gets 2-word provocative titles? The BBC is different, possibly because it has no commercial interests. It also opened its news API to the public recently, which was inarguably a wise step.
Some months ago, an automatic and effective way of obtaining music
