Learning new things with search in declining search relevancy

One of the things I love about search is the ability to stumble upon new topics of information outside of your designated area of interest.

For example, while looking through Google Images for an image of Rolls Royce designed jet engines for a reference story, I saw the following image and had to click through:

It’s from the site of Bruce Simpson, and his personal obsession with building pulsejet engines.

The design he uses has a basic enough premise - but it’s Bruce’s application and genuine sense of fun and exploration with the project that really comes through.

It’s the sort of site I don’t mind spending ten minutes of so from work reading up on - the subject is novel and extreme, and so can easily capture interest in the first place.

What is perhaps more of a shame is how relevance within search technology itself seems to be diminishing - especially on the Google search engine.

A number of times I have been looking for specific information from websites but completely unable to locate it - I’ve had to resort to using complex queries based on using the site:DOMAIN parameter to help narrow down requests to a list of pages where the information may be present.

Google used to be famed for its relevancy, and probably still is the most relevant search engine for most users in the UK at least (based on a Google claim of retaining a 70% share of the UK search market).

However, Google does seem to have reneged on its strengths by focussing less on finding the best content, and instead to penalising and devaluing least content.

While I can appreciate there are issues of “search engine spam” to deal with, search engines have always been faced with the problems of retrieving the most relevant information from a sea of documents - the difference ironically is that Google’s own AdSense program has made it profitable for many webmasters to enlarge the sea of documents.

Google’s approach to anti-spam issues is especially blighted by the fact that just because a site is new or aggressive in its promotion, does not equate with that site being irrelevant. In fact, the basic tenets of search engine optimisation revolve around accessibility issues, not least in helping search engines assign the relevant keyword meaning to the documents in question.

While this process has always been open to varying degrees of aggression, Google’s approach of suppression in my view is backfiring, and 2005 will remain the year that Google’s search relevancy declined, and entirely because Google has lost sight of the importance of relevancy to a user query, in lieu of fighting it’s own AdSense-powered sites out from the main index.

Two particular areas that seem to have suffered are phase-matching and authority - in the former, Google used to be great at relating different keywords to a page. However, now Google appears to place less emphasis on the direct relationship between such words in a query.

Additionally, Google has become horribly obsessed with “authority” issues, which effectively means a website can be regarded as more important for a wider range of queries simply of the basis of being older and more established.

For example, when search for an interview with Craig Silverstein at Slashdot for an official position on IP use, using the keywords “slashdot interview google engineer” has the Google Directory category for nanotechnology (a re-ordered DMOZ) listed third. Clearly, it has no relevancy for my untended query, and the only way it can be relevant to a user query is if someone were looking for information *from* Google, rather than *about* Google.

I’ve found the similar problem when searching for information from one site, only to see a completely different site listed first, because it is deemed a stronger “authority” - for example, Threadwatch over Platinax, and WebmasterWorld over SearchEngineWatch.

Of course, I could always use another search engine - there’s a lot about Yahoo! search I’ve found appealing recently, not least that it’s prime motivation still appears trying to distill best content from a general sea of content, as opposed to trying to punish other content first.

Microsoft’s recent venture into using neural networks has proved interesting from a technological point of view - but the reality of the situation as I observe it is that MSN has simply found a new way to elevate on-page elements to be the major ranking factor, and thus is in danger of developing 21st century technology to look like a 20th century search engine.

As a consumer, I’m becoming frustrated with the Google search brand, but in this young market, changing brands is not so difficult - Yahoo! search is now my homepage after 5 years of it being Google UK - and in the meantime, while I’ll be happy to stumble on novel and exotic sites, such as Bruce Simpson’s site on home-made jet-engines, search engines themselves need to be well aware of the consumer frsutrations they may be developing, not least by pitting their anti-spam tactics against their search relevancy.