Yahoo! indexing forces hosting account upgrade

I’ve mentioned before about how the Yahoo! slurp bots are really pounding at some of my forums, and so have some other vBulletin admins.

While improved Yahoo! deep indexing - as I suspect is the case in hand - is definitely to be applauded, it is also causing problems for some webmasters, including myself.

One of my forums has jumped from having around 40-80 visitors (members and guests) at any one time, to a regular figure of around 100-200 - most of which is populated by Yahoo! slurp bots.

This is causing such extensive resource usage that it’s overwhelming the SQL server with too many connections, resulting in the forums not always being properly responsive.

Additionally, I am now being advised by the hosting company to upgrade the forums in question: from a decent reseller account to a VPS account, especially as I am planning to extend the vBulletin forums to include an art gallery from Photopost.

Of course, search engines have advised in the past that they will only crawl dynamic pages more sparingly to preserve server load, but where mod_rewrite solutions exist - such as the vBulletin static archive - this can circumvent such safe-guards and cause problems for larger sites with strong traffic in the first place.

Still, what sort of webmaster whines about having too much traffic? And after recent problems with Yahoo! here at Platinax, I can only be grateful that the search engine at least respects some of my other sites. So a VPS account it is. :)