• Register
• Log in

SEO Theory - SEO Theory and Analysis Blog

Algorithm analysis, Web community relationship analysis, SEO practices and techniques, industry news, etc.

• Home
• About SEO Theory
• SEO Glossary

When Web sites vanish

Posted by Michael Martinez on January 25, 2008 in General

Do you ever check The Internet Traffic Report? I’ve been using this resource for years to solve mysterious Web site outages.

I’ve had more than one discussion with people who said they could not reach a site that I could reach, or vice versa. Whenever that situation arises usually one of two things is happening: the server hosting the site is in a problem situation or an Internet routing hub is experiencing difficulty.

ISPs route traffic through as many hubs as possible so that our Internet experience is as seamless as possible. ISPs enter into peering agreements with each other. Large ISPs with many network operations centers (NOCs) become backbones if they control land lines (or whatever communications pathways they use). The more bandwidth an ISP has going into a NOC, the more likely that NOC has a lot of peer connections.

Still, there are some major choke points. Whole states can lose track of the Internet while retaining their internal connectivity. In North America, Internet traffic passes through two major routing points — MAE East and MAE West — that carry unbelievable amounts of traffic every second.

If you and a client are unable to see your Web site, but your hosting ISP says everything is fine, try looking at the Internet Traffic Report or one of its competitors. If you see outages at key hubs you’ll know that you’re not going mad, there’s just a problem somewhere upstream from you.

Search engines often access Web sites we can’t see because they can get to those sites through pathways we don’t have access to. That’s one of the benefits of distributing a search service across multiple data centers. But sometimes the crawlers are disrupted by hub outages, too. That’s one reason why — sometimes — your pages may vanish from a search index temporarily.

Understanding how the Internet works — how the network as a whole continues to function while large sections of it go offline — helps you manage your anxieties, your clients’ anxieties, and your optimization strategy. You know that if a search engine cannot get to your site today it will be able to tomorrow — as long as the problem is just a typical connection point outage.