---

Getting Indexed in Google - The Googlebot

---

Pretty much everyone who is interested in search engine marketing has a vested interest in getting listed in Google. It is such an important search engine that we felt it worthwhile explaining how Google indexes sites and how you can tell if your site has been picked up in the latest Google trawl for inclusion in the latest search results.

There are two ways to get listed in Google. You can opt to submit your index page to Google directly here, or you can wait until Google spiders your site automatically, usually by finding your site via links on already listed sites. Once again, as we’ve discussed in other articles, you should actively seek to exchange links with other sites for a variety of reasons (link popularity being the key one). If you’ve just started out, links back to your new site will help the Google trawlers to find your site for the first time.

Google has two types of spiders – “Freshbot” and “Deepbot”. The Freshbot, as you’d expect from its name, is constantly crawling the web for updated information, so you may well get daily visits from it. The Deepbot goes out once a month and usually lasts a week – many search engine experts see the Deepbot as the more critical one, since it determines which pages get permanently added to the main index and looks at your link structure and therefore influences your page rank.

You can tell if “Googlebot” has visited your site by examining your own log files. You should see and entry such as this one, which indicates that either type of Googlebot has recently checked our your pages – it will appear as a “browser” in your records typically:

Googlebot/2.1 (+http://www.googlebot.com/bot.html)

There are also a wide variety of IP addresses associated with the Googlebot – the main ones you’d expect to see will be something like this:

64.*.*.* (Freshbot should start with this numbers)
209.*.*.* (Deepbot should start with this number)

If you have a new site, or indeed have added new pages to an already listed site, there are no firm guarantees as to when they will appear in the main index. Typically, you should wait a month to see your main page URL appear, and then several weeks after this your supplementary pages should start showing up.

If you have had no success after the first month or so, there are reasons why Googlebot didn’t like your site –

First off, try reading up on Google’s own tips:

“Occasionally, websites are not reachable when we try to crawl them because of network or hosting problems. When this happens, we retry multiple times, but if the site cannot be crawled, it will not be listed in our current index. If it was a transient problem, the site will likely show up in the next index, which will be completed in a few weeks.

If we have not picked up your site and it has been several months, then it is likely that our spiders are not able to find your site. If you increase the links pointing to the page, Google will likely find your site in the future.”

The most relevant section is My web pages are not currently listed. The main reasons would include:

Your web host was unreachable

You use dynamic pages

You use “Doorway Pages” (seen as spamming if used a lot)

Your site uses “Frames” (makes some sites harder to index)

Just bad luck – Googlebot may have had a technical glitch!

You have a robots.txt file in your directory root which is currently blocking Googlebot.

Posted April 25, 2005

Latest articles in Google Tips

 

Google Sandbox - new sites taking ages to get hits!
[March 24, 2006] Much has been written about the Google "Sandbox" theory, wherby new sites take many months before they receive search engine referrals. Some thoughts on this theory and links to further reading

 

Google manipulates search results: A boost for small business?
[March 11, 2006] Getting a Top 10 ranking in Google's search results is becoming harder by the day, especially for small businesses with limited budgets. But if a Top 10 ranking for your major keywords has eluded you so far, there may be hope for you.

 

Google Ban - How not to get banned by Google!
[February 6, 2006] Given that Google now provides over 70% of all Internet search traffic, the last possible thing any small business site owner would want is to be banned from the Google index!

 

BMW banned by Google for dubious SEO techniques!
[February 6, 2006] With so many small businesses now reliant on Google for a large chunk of their online traffic, the news that Google appears to have banned the German BMW site from its search results for dubious search engine marketing techniques just goes to show how important it is not to push your luck too far when trying to get high ranking for your keywords.

 

The sparring and spin of the Google dance
[December 21, 2005] The Guardian reports on how unscrupulous firms are manipulating world's leading search engine

 

SEO Duplicate Web Content Penalty Myth Exploded
[December 1, 2005] Discussion of the myth that search engines will penalise a site if much of its content is also on other websites.

 

Under the thumb of Google's Jagger Update
[October 20, 2005] As many of our newsletter subscribers and visitors take a keen interest in search engine ranking changes, I thought I'd write a few words about Google's latest update, nicknamed "Jagger".

 

“Google bombing” and what it can teach us about search engine optimisation
[September 29, 2005] Google bombing involves creating links to a certain page using the target keyword in the link. This is a legitimate method to use to improve search engine rankings, even if it is a method abused by the Google-bombers.

 

What is the Google Dance?
[August 31, 2005] Google is so critical to most site’s search optimisation process that entire sites have been set up purely to analyse how often, and precisely how Google updates its search results each month or so.

 

How long since Google updated Pagerank?
[August 22, 2005] According to one of our favourite web marketing tools (Page Rank Update List History), Google hasn't updates its 'Pagerank' or backlinks for 83 days.