• Register
• Login

SEO Theory - SEO Theory and Analysis Blog

Algorithm analysis, Web community relationship analysis, SEO practices and techniques, industry news, etc.

• Home
• About SEO Theory
• SEO Glossary

Site search optimization

Posted by Michael Martinez on October 17, 2007 in Advanced SEO

Site search is the 3rdGen Web navigation tool of choice. If you’re planning a large content Web site with at least 100 pages you really need to implement a site search feature from the beginning, and you need to make it as accessible as possible.

Site search is not easy to implement, much less to optimize for. You can slap a search box on your site from a major search engine but that is no guarantee that you’ll like what you get. And keep in mind that promoting one search engine over another on your site makes a very strong statement to your visitors.

CNN is probably the most popular news portal on the Web and CNN happens to use Yahoo! for site search. Google and Yahoo! claim to index about the same number of pages for CNN (although when you get up into the hundreds of thousands of pages you have to be careful about trusting those numbers).

You can compare the indexing depth of two search engines by looking for obscure content, such as a query for ‘michael martinez’ on CNN through Google and a query for ‘michael martinez’ on CNN through Yahoo!.

Yahoo! returned more results, but perhaps I’m not being fair to Google. After all, Microsoft returns 10 results for ‘michael martinez’ on CNN as well.

It’s interesting to see where the Business 2.0 Magazine table of contents shows up in their various search results. (NOTE: I am not the Michael Martinez whose name comes up in these queries.)

Restricting the relevance of a query to a specific site or section of a site both narrows and broadens the scope of your optimization. By eliminating competing sites from the search pool you ensure that only your desired content will be shown. But the rules by which you are accustomed to playing change.

For example, if you search Xenite.Org for ‘michael martinez’ through Google you’ll see my old SEO sub-domain come up first.

If you search Xenite.Org for ‘michael martinez’ through Yahoo! you’ll see my very, very ancient “homepage” come up.

If you search Xenite.Org for ‘michael martinez’ through Live you’ll see Xenite’s root URL come up.

My old “homepage” tells people to visit michael-martinez.com. My old SEO site tells people to contact 1st Query for SEO Services.

Microsoft’s algorithm made the best guess about which page is most relevant to my name and offered up Xenite’s root URL.

On the other hand, if you search SEO Theory through those three services Yahoo! and Live offer the root URL of the domain for my name and Google offers the blog URL (seo-theory.com/wordpress/). It’s a toss-up about which page is more relevant to me, but I wonder why none of the search engines offered the Michael Martinez profile page. It doesn’t even come up in Live’s first 10 results, although Yahoo! lists it 3rd for me and Google lists it 7th. Your results may vary.

Every post-migration post on this blog bears a link to my profile page using my name as the anchor text. No other page on the site has so many links pointing to it. No other page on the site has so many links with my name as anchor text pointing to it.

Those of you who are searching for information about the value of “internal links versus external links” might want to take a look at these queries and replicate them on your own sites. But before you conclude that external links outweigh internal links, think carefully about what we have learned through the past year.

Google does not allow links in its Supplemental Results Index to pass value to other pages — so far as I and other people have been able to determine, no anchor text goes out from any page that, in my opinion, is a Supplemental Page. My guesses about which pages go Supplemental may not be correct, but I suspect that more than half of SEO Theory’s pages are in the Supplemental Results Index.

Now, if that is true, then my Author profile page doesn’t have quite so much link anchor text pointing to it as one might normally conclude — not in Google’s algorithmic opinion. Those snarky Supplemental pages on SEO Theory just cannot be trusted to point you to the most relevant page about me on SEO Theory.

Well, Yahoo! has always said that the first link from any domain is given more weight than other links on that domain pointing to the same page. I can accept, therefore, that Yahoo! only places my profile page third after the two most highly linked URLs on the site.

Microsoft disappoints me, much though I have been impressed by their impressive performance over the past couple of weeks. The new Live.Com search is sending traffic to Web sites I once thought Microsoft would never care about. Still, my site search test implies that Microsoft may not be nearly as impressed with linkage as Google or Yahoo!.

Further testing is called for. Try to be discrete with your own tests, as I will be discrete with mine.

Now, I used these queries to illustrate the fact that the major search engines return different results for site searches. Before you just glibly add a search box from your favorite search engine to a Web site, you should spend considerable time running site searches to ensure that you use the best site search interface.

You want to make sure that your in-site architecture is solid because, if nothing else, the search engines need to crawl your pages. Google may throw a lot of pages into the Supplemental Results Index, which makes it difficult for those pages to rank well in a site search, but at least they’ll crawl and index the pages.

Yahoo! and Live tend to play more fairly with all your content in site search, so you need to evaluate which service’s choices most reflect your own thinking. You may be able to tweak your pages to come up in the order you wish through site search, and that in turn may help improve your relevance in normal Web search. But if you don’t see your most relevant pages in test queries on a particular engine that means you’re not giving that engine the signals it is looking for.

In any situation where two entities are interacting, problems can be in one of three states: they can be Entity 1’s responsibility, they can be Entity 2’s responsibility, or they can be the responsibility of both. It could be your site architecture is bad, it could be that the search service is bad, or it could be both.

Ideally you want to optimize for site searches on all four major search engines, but at the very least you want to optimize for Google, Yahoo!, and Live because they each offer greater indexing depth than Ask. That’s Ask’s choice to make but I’m not likely to ever use their service for site search. They’re just too picky.

I recently announced that I’m no longer using Google for Xenite’s site search because they are just too picky, too. I can understand Google not trusting Supplemental Results Pages in their general Web search, but why do they have to be so stingy with a site search? That makes no sense, except in the possibility that their algorithm is so paralyzed by fear and mistrust that it cannot perform optimally in what is substantially a very safe search environment.

No Supplemental Results Page is safe in the hands of Google search, and that is Google’s choice.

So, many of us need to choose beteen Yahoo! and Microsoft for our site search functions and the choices are not always easy to make. I, in fact, chose Microsoft mostly because I normally don’t get much traffic from them. When they started sending me more traffic than Yahoo! I decided the time had come to leverage Microsoft’s love for Xenite into site search.

I’ve used Yahoo! for site search in the past but after a certain point I feel their results drop off in quality. Maybe that has something to do with the fact that they suggest you search for “fandom wank” on Xenite even though they return no results for “fandom wank”. That just strikes me as weird. Yahoo! just doesn’t tend to separate the concept of site search from the concept of Web search as cleanly as I would like.

Nor does Google.

So that leaves Microsoft.

But having chosen a search service for site search is only the beginning of the process. You still need to optimize your content for site search. Otherwise, you’ll just frustrate your visitors.

Except for the fact that they occasionally suggest content on other sites, Microsoft does a pretty good job of connecting my visitors to my content. My Microsoft referrals have gone up since I changed site search functions, but I still have work to do.

As I renovate Xenite’s design, I’ll be emphasizing site search more and on-page navigation less. But that doesn’t mean I’m reducing on-page navigation. It just means I’ll be making site search more prominent. I hope that Microsoft is up to the task of figuring out the changes and keeping the right content at the top of site search results.

I’ve lost faith in Google and it will be a long time before I trust them with a Xenite site search again. My visitors need good site search, not just site search. There is a difference, and that difference begins with the choices I make for myself.

I’ll come back to site search optimization again in the future.