When's an Outage More than Just an Outage?
BIG BIZ By Aaron Shear, Search Engine Watch, Jul 29, 2008
Columns | Contact Aaron | Biography
I've written before about how site speed and performance is a major factor to search engines such as Google, especially when sites have an enormous amount of content.
It's difficult for a search engine to tell what the overall user experience is when someone clicks on a result, especially since they have so many possible combinations and new phrases that are used every day.
A search engine like Google can tell how much time a particular page takes to generate for a crawler. This time is taken as a factor in terms of a sites ability to produce a response in a reasonable amount of time.
Without knowing the exact formula, my view is just that, a view. It's safe to assume that Google uses an average of sorts per domain, which can calculate the amount of pages from a particular domain that can be hit at the same time. This is based upon the overall response time recorded during a crawl.
From what I can tell, sites that respond on average 500 milliseconds or lower tend to rank well within the Google search results. This, of course, is highly dependant on other ranking factors that are chosen. Additionally, I've seen that sites typically over one second on average tend to rank considerably less well at similar testing levels. This "demotion" tends to increase as the latency gets worse.
The same goes for a site, either large or small, disappearing for a period of time. In most cases, a search engine will see a sever based error message such as "500 server busy," a message that will be delivered with most throttling programs designed to keep a search engine from over crawling a site. A search engine such as Google will automatically assume that the Web server is overused and the users are seeing the same message.
Why would you want your user base to see this type of message while searching? Thus, in most common cases these results are removed from the index, until such problems are corrected.
During times of either configuration problems or simply an outage, a search engine may see one of the following: DNS is unavailable, Network is not available, 500s in a large volume, and if a server is setup incorrectly, it could see redirects to nowhere, all pages showing a 404 error (page or file not found), or it may just simply hang indefinitely without a timeout. Any of these problems can cause a site to fall out of the index, either very quickly or over time, depending on the situation or severity.
It's important as a large site to consider having multiple infrastructure backups. If you're dealing with more than 300,000 visitors a day, it may be necessary to have multiple data centers across your market. One of the best ways to ensure uptime is to use a service that caches your pages and keeps them hosted across the world, ensuring that a user or search engine can get to such data instantly. Panther Express is a very reliable service that has worked well for me over the past year.
It may also be beneficial to use a DNS service that will ensure your information is broadcast on a regular basis, thus if one of your servers goes out you will be covered (in most cases).
Make sure your operations team keeps your servers updated and online. Make sure they don't use a module such as "mod throttle" to keep spiders from overwhelming the servers. If they're hitting too hard, explore alternatives in your robots.txt files that the search engines can see. Or, in Google's case, go into your Webmaster Tools and turn down the crawl frequency from there.
» Print this article » E-mail a colleague » Send feedback
Biography
Aaron Shear is a partner in Boost Search Marketing, an enterprise-level global consulting firm. Offering expert advice to many of the most trafficked sites around the world.
Previously Aaron was the Global Director of SEO with Shopping.com an eBay Company. At Shopping.com Aaron spearheaded the global optimization efforts of Shopping.com, Dealtime.com and Epinions.com. Prior to joining Shopping.com, Aaron was the CTO at SEO Inc. where he spearheaded optimization efforts, with clients such as IGN Entertainment, VEGAS.com, Sierra Trading Post, Sony Motion Pictures, Archer Daniels Midland, Alliance Business Centers Network to name a few.
Before becoming an SEO Aaron worked at Inktomi, as a Technical Account Manager, where he learned SEO from the creators of the search engines first hand. Aaron's primary responsibility was managing client relationships such as MSN, IWon, Hotbot and HP to name a few.
Aaron has been optimizing websites since the late 90's, and has provided hundreds of businesses with countless top SEO and SEM returns.
Article Archives by Aaron Shear:
» When's an Outage More than Just an Outage? - July 29, 2008
» Recruiting SEO Talent for Big Sites - July 15, 2008
» Some New SEO Services Not So "Special" - July 1, 2008
» SEO Issues with Global Expansion - June 17, 2008
» When Top Keywords Suddenly Vanish - June 3, 2008
» Social Media Marketing in an Enterprise Environment - May 20, 2008
» More Articles by Aaron Shear
We want to know what you think about Aaron Shear’s column, "When's an Outage More than Just an Outage? ". Rant. Rave.
» Voice your opinion
No comments:
Post a Comment