Brian Turner's Business Blog
 
Business, Marketing, Search, Internet, Blogs, Forums, and Tech
March 27, 2006

Google spider *does* find content without links

It’s something I’ve seen suggested on-and-off for a couple of years - both that the Google spider will visit URLs via the Google Toolbar, along with a claim that this may help facilitate spidering.

I’m seeing proof of spider activity on a closed site now - which answers the question quite firmly for me.

I’m setting up a forum on one of my newer established sites - while the main site has a few links, there are absolutely no links to that forum.

Today the following IP is visiting the forum: 66.249.72.194 - and it’s identified as a Google spider.

I’m watching it now and how it behaves - it’s checking out the various links on the page, and it’s really interesting to watch live a Googlebot sniffing around someplace it’s never been openly invited to visit.

I am using the Google Toolbar to surf it in IE for error checking, so my primary suggestion is that the Google Toolbar does indeed invite the spiders in.

However, I can’t help but wonder what sort of data collection activity may be happening with Mozilla Firefox, my browser of choice.

Either way, I’ll kep the forum closed from public view while I continue to develop it, and see whether GoogleBot actually indexes any of the URLs it finds.

ADDED: The spider also indexed 8 links from the main page, which are now recorded on Google.



Related posts to:
"Google spider *does* find content without links":



1 Comment »
  1. […] I also reported a couple of weeks ago that another new forum I was developing on a domain was being indexed by the AdSense mediabot. […]

    Pingback by Brian’s Business Blog » Google loses its index — April 17, 2006 @ 8:43 am

Leave a comment


Previous: « The Dangers of Drink Driving
Next: 40 Best Web Directories »

Visited 583 times, 1 so far today since July 24th 2007