Handel » Each of the engines

There has been endless webmaster speculation and worry about the so-called "Google Sandbox" - the indexing time delay for new domain names - rumored to last for at least 45 days from the date of first "discovery" by Googlebot. This recognized listing delay came to be called the "Google Sandbox effect."


Ruminations on the algorithmic elements of this sandbox time delay have ranged widely since the indexing delay was first noticed in spring of 2004. Some believe it to be an issue of one single element of good search engine optimization such as linking campaigns. Link building has been the focus of most discussion Mike Alstott Buccaneers Jersey , but others have focused on the possibility of size of a new site or internal linking structure or just specific time delays as most relevant algorithmic elements.


Rather than contribute to this speculation and further muddy the Sandbox, we'll be looking at a case study of a site on a new domain name, established May 11 Matt Gay Youth Jersey , 2005 and the specific site structure, submissions activity, external and internal linking. We'll see how this plays out in search engine spider activity vs. indexing dates at the top four search engines.


Ready? We'll give dates and crawler action in daily lists and see how this all plays out on this single new site over time.


* May 11 Anthony Nelson Youth Jersey , 2005 Basic text on large site posted on newly purchased domain name and going live by days end. Search friendly structure implemented with text linking making full discovery of all content possible by robots. Home page updated with 10 new text content pages added daily. Submitted site at Google's "Add URL" submission page.


* May 12 - 14 - No visits by Slurp, MSNbot, Teoma or Google. (Slurp is Yahoo's spider and Teoma is from Ask Jeeves) Posted link on WebSite101 to new domain at


* May 15 - Googlebot arrives and eagerly crawls 245 pages on new domain after looking for Mike Edwards Youth Jersey , but not finding the file. Oooops! Gotta add that file!


* May 16 - Googlebot returns for 5 more pages and stops. Slurp greedily gobbles 1480 pages and 1892 bad links! Those bad links were caused by our email masking meant to keep out bad bots. How ironic slurp likes these.


* May 17 - Slurp finds 1409 more masking links & only 209 new content pages. MSNbot visits for the first time and asks for 75 times during the day, but leaves when it finds that file missing! Finally get around to add by days end & stop slurp crawling email masking links and let MSNbot know it's safe to come in!


* May 23 - Teoma spider shows up for the first time and crawls 93 pages. Site gets slammed by BecomeBot, a spider that hits a page every 5 to 7 seconds and strains our resources with 2409 rapid fire requests for pages. Added BecomeBot to exclusion list to keep 'em out.


* May 24 - MSNbot has stopped showing up for a week since finding the file missing. Slurp is showing up every few hours looking at and leaving again without crawling anything now that it is excluded from the email masking links. BecomeBot appears to be honoring the exclusion but asks for that file 109 times during the day. Teoma crawls 139 more pages.


* May 25 - We realize that we need to re-allocate server resources and database design and this requires changes to URL's Jamel Dean Youth Jersey , which means all previously crawled pages are now bad links! Implement subdomains and wonder what now? Slurp shows up and finds thousands of new email masking links as the was not moved to new directory structures. Spiders are getting errors pages upon new visits. Scampering to put out fires after wide-ranging changes to site, we miss this for a week. Spider action is spotty for 10 days until we fix


* June 4 - Teoma returns and crawls 590 pages! No others.


* June 5 - Teoma returns and crawls 1902 pages! No others.


* June 6 - Teoma returns and crawls 290 pages. No others.


* June 7 - Teoma returns and crawls 471 pages. No others.


* June 8-14 Odd spider behavior, looking at only.


* June 15 - Slurp gets thirsty Sean Murphy-Bunting Youth Jersey , gulps 1396 pages! No others.


* June 16 - Slurp still thirsty, gulps 1379 pages! No others.


So we'll take a break here at the 5 weeks point and take note of the very different behavior of the top crawlers. Googlebot visits once and looks at a substantial number of pages but doesn't return for over a month. Slurp finds bad links and seems addicted to them as it stops crawling good pages until it is told to lay off the bad liquor, er that is links by getting to slap slurp to its senses. MSNbot visits looking for that and won't crawl any pages until told what NOT to do by the file. Teoma just crawls like crazy Devin White Youth Jersey , takes breaks, then comes back for more.


This behavior may imitate the differing personalities of the software engineers who designed them. Teoma is tenacious and hard working. MSNbot is timid and needs instruction and some reassurance it is doing the right thing, picks up pages slowly and carefully. Slurp has addictive personality and performs erratically on a random schedule. Googlebot takes a good long look and leaves. Who knows whether it will be back and when.


Now let's look at indexing by each engine. As of this writing on July 7 O. J. Howard Youth Jersey , each engine also shows differing indexing behavior as well. Google shows no pages indexed although it crawled 250 pages nearly two months ago. Yahoo has three pages indexed in a clear aging routine that doesn't list any of the nearly 8,000 pages it has crawled to date (not all itemized above.) MSN has 187 pages indexed while crawling fewer pages than any of the others. Ask Jeeves has crawled more pages to date than any search engine, yet has not indexed a single page.


Each of the engines will show the number of pages indexed if you use the query operator "site: " without the quotes. MSN 187 pages Vita Vea Youth Jersey , Ask none, Yahoo 3 pages, Google none.


The daily activity not listed in the three weeks since June 16 above has not varied dramatically Mike Evans Youth Jersey , wHave you ever thought a. Cheap Jerseys From China Cheap Jerseys China Free Shipping Cheap Soccer Jerseys Cheap NHL Jerseys Cheap NFL Jerseys Wholesale Cheap Authentic Jerseys Cheap Replica Jerseys Cheap NFL Jerseys Wholesale Cheap Authentic Soccer Jerseys Cheap Hockey Jerseys