The Wardman Wire

You are browsing the archives of yahoo spider.

This week I have started using a Wordpress plugin called “Crawl rate Tracker”.

The plugin draws a graph of how often your website has been visited by the indexing spiders from Google, Live Search (Microsoft), Yahoo and Technorati. This is important since this is an indicator of how much trust particular pages attract from a search engine.

The graph below is from the Wardman Wire for the few days since I installed the plugin. As you can see, the blog has had up to 1000+ search engine spider visits each day - which makes clear why it is important to filter “robots” out of your traffic statistics. Also, the front page is being indexed up to 13 times a day by Google.