• Register
• Log in

SEO Theory - SEO Theory and Analysis Blog

Algorithm analysis, Web community relationship analysis, SEO practices and techniques, industry news, etc.

• Home
• About SEO Theory
• SEO Glossary

Designing the perfect SEO tool

Posted by admin on December 13, 2006 in SEO Theory

Search engine optimizers love their tools, and for the life of me I don’t know why. Most of the tools I have seen through the years are pretty much duds and losers. The basic SEO tool these days is some sort of backlink checker. If you’re “lucky”, you’ll also get Toolbar PR reports for each link.

Of course, as everyone knows, Google won’t tell you about all the backlinks it has indexed. So the amateur SEOs immediately tell you, “No problem! Just check your backlinks on Yahoo!”

But there is a problem because Yahoo! doesn’t share data with Google. Yahoo! only knows about the pages Yahoo! has crawled and Google only knows about the pages Google has crawled. Through the years, I’ve tracked batches of dozens of Web sites as the search engines indexed them. Getting 20-40 sites indexed in Ask, Google, Live, and Yahoo! takes time. It usually takes up to 3 months before the first site appears in Ask, and oftentimes longer. In the meantime, your Yahoo! column is riddled with holes, your Google column is riddled with holes, and you cannot line them up.

If you ever want to know which side is outgunned in an SEO shootout, just shut up and see who is the first to start talking about “PR”, “checking backlinks on Yahoo!”, and “the great SEO tools” they use. There’s your amateur SEO link-slinger, waiting to get the pants shot off him (or her) by a more seasoned hand.

Of course, like all good (and not-so-good) SEOs I occasionally check backlinks. I don’t do it because I believe knowing what Yahoo! knows about will tell me how I’ll rank for any given expression. I do it so I have an idea of which pages pass value, which pages are likely to be crawled next, and which pages are orphaned (they have no value-passing links pointing to them).

Neither Yahoo! nor Google will give up their link-analyzing secrets, but you don’t need to know what the search engines think to develop a decent backlink checking tool. You just need to know what they’ve indexed. Fortunately, you can find out what all four major search engines have indexed. Just run URL queries. Ask the search engines, “Do you know about this URL?” They’ll tell you in one way or another.

So how does that help with backlinks? Let’s take a typical SEO backlink checking tool. Let’s call it the Stupid Backlink Checking Tool (or SBCT for short). SBCT dutifully goes to Yahoo! and says, “Dear Yahoo! Site Explorer — what are the backlinks for this URL?”

Many SEOs don’t realize that SBCT is getting bad information even for Yahoo! because you can find pages indexed in Site Explorer that don’t show up in Yahoo!’s main index. But that’s okay because we can still go through each of SBCT’s backlinks (let’s say there are only 300 of them) and see which ones show up in Yahoo!’s main index. If we’re lucky, maybe 250-280 of the links appear in Yahoo!’s main index.

That doesn’t tell us if those links pass value for Yahoo!, but we can filter the links by domain name and assume that each domain only passes value for one link (Yahoo! hints this is closer to the truth but they don’t say exactly what they are doing). Let’s assume we end up with 150 backlink domains showing in the main Yahoo! index.

That’s not bad but it doesn’t tell us whether any of the links pass value. Nor does it tell us what links Ask, Google, and Live know about. But we have 150 unique backlinks that may be helping with our performance in Yahoo!.

Now, let’s run a URL reference query at Google. A URL reference query is posed in the form of “example.com/”. You can change it up a little, such as going for “www.example.com/” or “exaple.com/index.html” (I would not do that except to find out who is linking to my index page improperly).

Google will happily tell you who refers to that URL in their indexable content. The references may be links and they may simply be textual representations of the URLs. But here is the good news: you can run the same query at Yahoo! Yes, you can actually compare apples-to-apples with Yahoo! and Google (or something akin to that kind of relationship). So now you can compare your Google URL reference results to your Yahoo! URL reference results. Let’s say you get about 340 on Google and 297 on Yahoo!

What then? Well, you have two sets of URL references, so to make any sort of valid analysis about both search engines you need to find the union of those sets. That is, you need to find all the URL reference hits that Yahoo! and Google have in common. Let’s say you come up with 213 shared URL references. You’re getting close to some useful information, but you’re not there yet.

Now compare your URL references to your Yahoo! Site Explorer backlinks. That’s how you weed out the links from the non-links for Google. It doesn’t give you all the links Google knows about, but it tells you how many links both Google and Yahoo! Site Explorer know in common. Often, the same comparison between Google and the filtered Yahoo! backlinks (of which we assumed for this example SBCT found about 250-280) will produce a smaller number.

So, if you have 213 common links between Yahoo! Site Explorer and Google, you may have only 180 common links between Yahoo!’s main index and Google. And then your 150 Yahoo! linking domains may have just gone down to about 135-140. But that’s okay.

You’ve just found 180 backlinks in Google. But that was a lot of work, and Stupid Backlink Checking Tool really hasn’t told you yet which links are passing value. It’s almost guaranteed — especially if you’ve been building links — that many of them do not pass value for any number of possible reasons. But let’s assume that 90% of your backlinks really do pass value.

How do you prove that? It’s impossible to do by looking at backlinks, but since you’ve got 180 common links you might as well look at these pages and see where they are linking to. Assuming your page is devoted to a reasonably popular topic, the odds are pretty good that many of those 180 links appear on pages that point to similar pages. Those 180 pages may link out to about 1800 other pages.

If you just eyeball the list of 1800 pages, looking only at their URLs, their title elements, and maybe their meta description tags, you and SBCT may be able to eliminate any obvious spammy sites. Let’s say you kill about 18 of your 180 links for pointing to obvious spam. Now they may be passing value, but you’re having to be conservative in this game because Stupid Backlink Checking Tool won’t tell you anything useful.

Still you’ve got 162 links left that may pass value in both Google and Yahoo!. So now what?

While I do have some shortcut tests I use to evaluate links, they would not be practical for use in a Stupid Backlink Checking Tool. Either you take your 162 links and assume they pass value or you take some risks and make some guesses.

Risk number 1: See if these links are in Ask. Ask does not like scraping, so I would not advise you to do it. They’ll ban your IP address. But assuming you can figure out how to check 162 linking pages in Ask without getting banned, it’s almost guaranteed that any pages which you find there will pass value in Yahoo! and Google. Not quite, but if Ask trusts a page you can bet the other search engines are very likely to trust it, too.

Risk number 2: Assume that each of your 162 backlinks passes value. All 162 pages help whomever they link out to. If you can manage to get some more links from those very pages, you may be doing well. But that is lumping all your eggs into one basket.

Risk number 3: With 162 inlinking pages, you ought to be getting some value. Create a link back to one of your 162 backlinkers and pass it some unique anchor text. Doing this one page at a time would take a while — it’s not practical. You’d have to link back to a lot of pages at one time (which won’t hurt you, but I’ll save the PageRank leaking nonsense for another time).

Let’s say you link back to 20 pages. What’s your risk? That you’re linking to 20 bad neighborhoods. Oops! You might just find your page is delisted. Sorry.

But if your 20 targets receive the unique anchor text you pass to them through your links in Yahoo! and Google, odds are pretty good they pass value to you. Congratulations. You’ve just identified 20 sweet value-passing linking pages.

It should take from 2 weeks to a month to perform such a test. Obvisouly, if you really have 162 inlinks, qualifying them in this way would take a great deal of time. Of course, if you have more than 1 value-passing page at your disposal (say, 10 of them) you could vet all 162 linking pages in about a month.

Clearly, no SEO tool running on someone’s Web server can tell you what is happening with links. There are good reasons to want to know who is linking to you and whether they are passing value, but the truth is that it usually doesn’t matter. Once your pages receive value from other people’s pages, and you get enough inbound links to stay in the main indexes, you’re pretty much done with your link-building.

If you have optimized your content properly, you can compete for most commercial expressions with fewer than 100 inbound links. As long as they are trusted links that pass value, that is. After all, 1000 spammy links won’t pass value, so if you’re building links in volume don’t count on getting much bang for your buck (although it should help with your crawling right up until you get delisted).

Does that mean all SEO tools are useless? No, not totally. In fact, for some limited purposes, I find a few tools out there help in small ways. For example, Jim Boykin recently published several new free SEO tools. I like the Strongest Subpages Tool because it can help guide people in developing their internal linkage. I also like the Forward Link Title Tag Tool and the Internal Link Title Tag Tool.

I can pretty much do without the rest of the tools as they won’t tell you anything useful for search engine optimization. They’ll give people who believe the various SEO myths warm fuzzy feelings, so if you’re anxious about your rankings I guess having a binky tool to make you feel good is worth something.

The SEOmoz PageStrength Tool is a better substitute for people who have to see a number between 0 and 10. It relies too much on Yahoo! without being able to do any functional analysis, but it at least looks at other factors.

Rex Swain’s HTTP Viewer can also be useful as well as various spider simulator tools. You might also get some interesting info from SEO Browser.

What you do with the information you find is your business, but another great myth in the SEO world is that it’s easy to do competitive analysis. The truth is that most SEOs don’t seem to do a very thorough competitive analysis. The majority of people are satisfied with looking at (Yahoo!) backlinks, PR, and keywords (often using keyword extractors). That information tells you virtually nothing useful about a competitor’s performance.

Maybe one day we’ll go down that road and see why competitive analysis is something else altogether.