• Register
• Login

SEO Theory - SEO Theory and Analysis Blog

Algorithm analysis, Web community relationship analysis, SEO practices and techniques, industry news, etc.

• Home
• About SEO Theory
• SEO Glossary

SEO Experiments Anyone Can Do

Posted by Michael Martinez on April 24, 2008 in SEO Theory

SEO experiments are an important part of the learning process in SEO theory but as we have no industry standards people don’t agree on how to structure SEO experiments. An SEO experiment should test a hypothesis OR it should generate data for analysis. Sometimes you can accomplish both tasks but I think that people who are curious about how I would structure SEO experiments should see some simpler types first. If there seems a lot of interest in this post I’ll write up some more experiments over the next week or two.

SEO Experiment 1: Does Toolbar PR correlate with link: queries?

This experiment helps you build your analytical skills.

Toolbar PageRank, which Mike Grehan famously dubbed “green fairy dust”, takes a beating on a daily basis from the SEO community. Nonetheless, every time Google pushes out new Toolbar PR data all the forums light up with threads and Barry Schwartz feels obligated to report on the discussion activity at SE Roundtable. Until last fall (2007), when Google deliberately reduced Toolbar PR values for many sites they concluded were selling links, there seemed to me to be no real value in tracking Toolbar PR data. However, PR fans can explore some of the deeper mysteries of the Toolbar without looking too foolish.

Although Google has never disclosed how they calculate Toolbar PageRank, Matt Cutts says that Toolbar PageRank data is only published 3-4 times a year whereas internal PageRank is calculated continuously by dedicated PageRank computing machines (Ibid.). And while some people insist that Toolbar PageRank is a logarithmic scale of internal PageRank, Google’s documentation suggests that more than internal PageRank is used.

So even if the Toolbar PR is not solely derived from linkage, it does seem to many people there is a very strong connection to linkage. That means we can postulate there is a correlation between Toolbar PageRank and value-passing links. Hence, we should be able to test that assertion. But to test the assertion we need to measure value-passing link data; can we measure value-passing link data in Google?

Yahoo! is no help for this experiment. We have to use data that only Google can (and will) provide, which leaves us with the link: query operator — that Google has confirmed only displays a random selection of links (Webmaster Tools is not an acceptable alternative for too many reasons to include here). For the sake of this experiment, it is okay to assume that the link: query operator only reports link sources whose links pass value (anchor text and/or PageRank).

What we need: A spreadsheet, 100 randomly chosen Web sites, the Google Toolbar (no substitutes), approximately 6 months of time.

What we do:

1. Record the Toolbar PageRank for all 100 randomly chosen Web sites
2. Record the number of listings reported by Google’s link: query operator for all 100 sites
3. Wait for the next Toolbar PR update (SEO forums and blogs will explode with PR update discussions)
4. Record the Toolbar PageRank for all 100 sites again
5. Record the number of listings reported by Google’s link: operator for all 100 sites again
6. Wait for the next Toolbar PR update
7. Record the Toolbar PageRank for all 100 sites again
8. Record the number of listings reported by Google’s link: operator for all 100 sites again

Admittedly it takes a lot of patience to perform this experiment but we need three capture points for every Web site. Since we’re seeking a correlation between Toolbar PR and reported links we need to capture two data items at each capture point. Now that we have three captures for our 100 randomly selected sites we can compute two vectors for all the sites. Vector 1 will show us the change (or lack of change) in Toolbar PR for a site. Vector 2 will show us a change (or lack of change) in the number of reported links for the site.

Now we want to look for consistency in the data relationships. If all or a majority of the vector paths follow similar paths across their three capture points, we’ve identified a trend (and therefore a correlation) relationship between Toolbar PageRank and Google’s randomly link selections.

SEO Experiment 2: How thoroughly are new Web sites indexed by Google?

This experiment helps you build your indexing skills.

What we need: A new Web site that has not yet been indexed, 2 paragraphs of simple text, and 2 paragraphs of sophisticated text.

What we do:

1. Create two pages for our new site
• The first page uses 2 paragraphs of simple text (no more than 3 syllables in any word)
• The second page uses 2 paragraphs of sophisticated text (include at least 20 words with 4-5 syllables)
2. Embed a link on each page that points to a relatively obscure site we think may be Supplemental
3. Add both pages to our site
4. Link to both pages from the root URL of our site (or whatever passes for our main page)
5. Wait 2-4 weeks for the pages to appear in Google’s search results

As with so many SEO experiments this one also requires some patience, especially since we don’t know how long it will take for Google to find and crawl the new pages. In my recent experience, new Web sites with XML sitemap submissions generally appear in Google within 1-2 weeks (but Google does not guarantee results, so neither can I).

Now, when we see both pages appear in a site: query, grab 5 words from the simple paragraph page and search for them (within the site search results). Surround the words with quotes (for an EXACT FIND mode query). Does Google show us any results? If not, what can we conclude about how thoroughly Google indexes new pages? What does it take to get a page fully indexed in Google?

Now let’s perform an EXACT FIND mode search for the anchor text in our outbound link. There are four possible results:

1. No pages are listed
2. Only the linking page is shown (no destination)
3. The linking page appears BEFORE the destination page
4. The linking page appears AFTER the destination page

What do each of these outcomes tell us about how Google indexes new page content and links? What can we deduce if we compare the results of the first query with the results of the second query?

Now let’s grab 5 words from our second page. We want to include at least one of our complex words in the 5. Then perform an EXACT FIND mode query (with quotes around the expression). If our multisyllabic words are relatively rare, there’s a good chance that at least one of them is indexed. In fact, we can perform a site: search for any of those words to see if that is the case.

Do we see different results for this query than we saw for the first query on the first (simple words) page?

Now let’s perform an EXACT FIND mode search for the anchor text in our outbound link on the second page. Do the results differ from the result of the first page’s anchor text query? If one of the multisyllabic words is included in the outbound anchor text, does that appear to make a difference?

Variations on this experiment: We can do this experiment on all the major search engines. We can also perform the experiment with multiple Web sites. It would be interesting to see how 20 distinct sites perform compared to each other.

Use experiments to learn and to train

Here at 1st Query, we use experiments, both simple and complex, in our regular search engine optimization process. We use the experiments to analyze algorithmic behaviors (which is entirely different from trying to figure out how rankings are determined) and we use experiments to train employees in fundamental SEO principles. An experimental Web site can be used over and over again to test ideas and assumptions. In fact, if members of my staff confront me with assumptions they’ve either made or acquired from reading SEO blogs, I ask them to create Web sites to test the assumptions (not that they always do so, but it’s the easiest way to convince me you’re on to something).

There is no standard set of SEO experiments but there should be. Generally speaking, if you can structure an experiment around site search it should translate well to general Web search. But if you’re interested in analyzing Blog Search, News Search, Image Search, etc. then the site search methodology doesn’t help you. You have to develop other methods for experimenting.

An experiment should not be designed to prove a concept. That is, simply testing a blunt assertion (links improve rankings, too many links result in penalties, etc.) is not as simple as most people in the industry think it is. Just to prove that links improve rankings requires that you factor out all other possible influences on rankings. Intuitively we are pretty sure that links can influence rankings, and there is even a large body of technical and informal literature that supports this intuitive understanding, but no one has yet shown how to prove this in SEO tests.

It’s better to design an SEO experiment that provides you with a lot of results. When you organize those results, you should be able to identify trends in the data patterns. The trends will emerge from cause-and-effect relationships so your experiments can grow in complexity and depth, reinforcing your better conclusions and disproving your incorrect deductions. A good SEO experiment is conducted in stages.

There are, of course, very simple SEO experiments that can produce impressive results quickly. Russ Jones took the SEO community by storm with his Virante sub-domain test in 2006 (the search results were presented so that when you read the sub-domains from top to bottom you saw a coherent marketing message). But these types of experiments don’t provide much information that is helpful in real-world search optimization.

Ask your new employees to perform experiments. Help them design some experiments until they become comfortable with the process and start proposing their own. The more you experiment, the more proficient you become at asking what it is the experiment can show you. In my experience, the more precise and definitive the answer to “What is this experiment going to show me?” turns out to be, the less useful and informative the experiment is. If you’re demonstrating a principle, the experiment is helpful. If you’re looking for insight into search optimization, knowing the answer in advance of the experiment doesn’t teach you anything new or help you grow.