Monday, June 27th, 2005, 5:00 pm
Googlebot versus MSNBot

Bill Gates arrested in his younger days (photo in public domain)
n item I have just come across (hat tip Justin Moore) speaks about excluding MSNBot based on poor performance. MSNBot (and Yahoo! likewise) consume a great deal of bandwidth, but are barely used as search engines by the Internet audience.
From Justin:
- Exhibit A – MSNbot Crawl hits = 9561
- Bandwidth used = 124.43 MB
- Visits to wantmoore.com as a results from searches @ Google = 683* [sic.*]
- Exhibit B – Googlebot Crawl hits = 3415
- Bandwidth used = 51.74 MB
- Visits to wantmoore.com as a results from searches @ MSN = 41 [sic.*]
…
So, effective immediately:
User-agent: MSNBot
Disallow: /* Order of the visits is wrong and should be reversed.
I spotted similar statistics on my domain, with Google as the top referrer at almost 20,000 visits/month. MSN remain at a miserable 200 and take almost as much bandwidth as Googlebot. I have plenty of bandwidth to spare; So, I might as well let Microsoft spend crawling time in vain.






Filed under: 
OV-Ray, an Open Source 3-D rendering package that could compete with the commercial Maya, 
igh-resolution satellite pictures of the States have been available for quite a few months. These have now been extended, by 


Scientists in Japan are developing equipment that can enhance human strength. Awkward as it may seem, such developments might have a positive impact. Robotic assistance to one’s muscles means that more elderly and disabled people will walk up the stairs without great difficulties. One the other hand, when power suits like that get misused, they can make us appear like bunch of supermen, powered by batteries. This
I am providing