Google spider *does* find content without links
It’s something I’ve seen suggested on-and-off for a couple of years - both that the Google spider will visit URLs via the Google Toolbar, along with a claim that this may help facilitate spidering.
I’m seeing proof of spider activity on a closed site now - which answers the question quite firmly for me.
I’m setting up a forum on one of my newer established sites - while the main site has a few links, there are absolutely no links to that forum.
Today the following IP is visiting the forum: 66.249.72.194 - and it’s identified as a Google spider.
I’m watching it now and how it behaves - it’s checking out the various links on the page, and it’s really interesting to watch live a Googlebot sniffing around someplace it’s never been openly invited to visit.
I am using the Google Toolbar to surf it in IE for error checking, so my primary suggestion is that the Google Toolbar does indeed invite the spiders in.
However, I can’t help but wonder what sort of data collection activity may be happening with Mozilla Firefox, my browser of choice.
Either way, I’ll kep the forum closed from public view while I continue to develop it, and see whether GoogleBot actually indexes any of the URLs it finds.
ADDED: The spider also indexed 8 links from the main page, which are now recorded on Google.
Previous: « The Dangers of Drink Driving
Next: 40 Best Web Directories »
Visited 583 times, 1 so far today since July 24th 2007



[…] I also reported a couple of weeks ago that another new forum I was developing on a domain was being indexed by the AdSense mediabot. […]
Pingback by Brian’s Business Blog » Google loses its index — April 17, 2006 @ 8:43 am