• Register
• Log in

SEO Theory - SEO Theory and Analysis Blog

Algorithm analysis, Web community relationship analysis, SEO practices and techniques, industry news, etc.

• Home
• About SEO Theory
• SEO Glossary

Why SEOs need crawl pages

Posted by Michael Martinez on March 3, 2008 in Intermediate SEO

Crawl pages have found a lot of uses through the years. Spammers use them to get their spam indexed. A lot of Web sites adapted the crawl page format to HTML sitemap structures (incorporating appropriate page design to make the sitemaps look like the rest of the site). But old-school SEOs used to use crawl pages to help the various search engines find their link source content.

For example, you could search Altavista for pages that linked to your pages, compare that query to results from an Inktomi search engine, and then build a crawl page that linked to pages not showing up in Inktomi. Finally, you’d submit the crawl page to Inktomi — better yet, you would link to the crawl page from pages you already had indexed in Inktomi and let Inktomi find those pages pointing to your site through its crawling.

It worked like a charm. In fact, it worked so well people did this cross-linking crawl-page thingee for all the major search engines. But people don’t do that so much, now.

There is nothing wrong with linking to the pages that link to your site. In fact, if you find yourself cringing at the thought of linking to pages that link to your site, then you’re thinking about the whole business in the wrong way. You should write an article about the pages that link to your site or create an elite directory about the pages that link to your site and tell your visitors about those other sites.

If you find yourself cringing at the idea of telling your visitors about pages that link to your sites then maybe you got your links from the wrong kind of pages.

Today you have Ask, Google, Live, and Yahoo!. They don’t all index the same content. Or, rather, they all index some of the same content but they each index content that the others don’t index. Hence, the old crawl page trick can still work for you with today’s search engines — provided you link to sites that are worth telling people about.

Linking to links you build is okay as long as you can do so in a context that makes those linked links useful and informative. There are links and there are links, but the best links exist because someone cared enough to create them even if they don’t actually point to anything of value. That is, if someone creates a Web directory listing all their favorite corn dog recipes, they may be linking to content you don’t care about but they created some great links for those corn dog recipe pages.

It really doesn’t matter if the search engines index the list of links to corn dog recipes if the list is where people can find it and follow the links. But if the list hasn’t been indexed and you just happen to have a corn dog recipe page (or 10) that you found on the list, why not help the search engines find the list?

Is this reciprocal linking? Yes.

Is this against search engine guidelines? No.

Can it be abused? Absolutely. You could create 1,000 crawl pages on as many domains. Hm. Would that leave a footprint large enough for a spam filter to be tripped? I don’t know. Why don’t you try it out and let me know? Or maybe you should just use your common sense and only create enough crawl pages to take care of the basic need to tell people about interesting content that links to your content.

People create crawl pages every day. You see them on SEO blogs all the time. “Here are 10 great links to Web sites I like!” That’s a crawl page. It’s not ugly like a spammer’s bare-bones, nothing-but-links crawl page but it’s still a crawl page. Just because you’re a human being doesn’t mean you aren’t crawling the Web, too.

SEOs love their top ten and weekly link lists. They push content all the time. It’s a tried-and-true function that falls under “best practices” SEO without all the publicity that goes along with creating good titles, page URLs, and navigational link anchor text.

If you can justify creating a page of links without breaking a sweat, you can create a crawl page. If you’re going to create a crawl page, why not create one that ultimately helps you by pointing search engines to content they may not already have found? You may think you only need traffic from Google but, you know what? That’s probably because you don’t know how to get traffic from the other search engines.

Google’s real market share is, at best, probably no more than 40% of the total market. Over 100 million people visit Yahoo! and over 100 million people visit Microsoft’s MSN and Live every month. Tens of millions of people visit Ask every month. If you’re not getting traffic from those search services, maybe it’s just because you don’t care enough about your search visibility to do something useful.

In fact, if you’re putting all your search eggs into Google’s basket you’re asking for trouble. Sooner or later you’re going to lose that Google traffic and then you’ll wish you were getting traffic from Yahoo!, Live, and Ask. I’d rather be the guy who doesn’t have to care about Google than the guy who hopes he never falls out of Google.