Googlebot versus MSNBot

Bill Gates arrested in his younger days (photo in public domain)
n item I have just come across (hat tip Justin Moore) speaks about excluding MSNBot based on poor performance. MSNBot (and Yahoo! likewise) consume a great deal of bandwidth, but are barely used as search engines by the Internet audience.
From Justin:
- Exhibit A – MSNbot Crawl hits = 9561
- Bandwidth used = 124.43 MB
- Visits to wantmoore.com as a results from searches @ Google = 683* [sic.*]
- Exhibit B – Googlebot Crawl hits = 3415
- Bandwidth used = 51.74 MB
- Visits to wantmoore.com as a results from searches @ MSN = 41 [sic.*]
…
So, effective immediately:
User-agent: MSNBot
Disallow: /* Order of the visits is wrong and should be reversed.
I spotted similar statistics on my domain, with Google as the top referrer at almost 20,000 visits/month. MSN remain at a miserable 200 and take almost as much bandwidth as Googlebot. I have plenty of bandwidth to spare; So, I might as well let Microsoft spend crawling time in vain.






Filed under:
I am providing
Some months ago, an automatic and effective way of obtaining music

ne issue is particularly detrimental to OSS: the issue is branching. Implementation takes different directions (often due to third-parties involved) and compatibility begins to break. Linux and its distros are one valid example. Here is an example taken from skinning (AKA themes) to support the argument.
discovered this
The hard-drive peripheral in for the Playstation 3 will run Linux. This news is by no means surprising as Sony’s direct opponent in the realm of home gaming is Microsoft. This