Search Engine Optimization: How search engines works

by Ing. Roberto Grassi

GRSoftware President

Introduction

Some studies made about search engines say:

The latest estimate of web pages on the World Wide Web: more than 8 billion and growing.
People who spend five or more hours a week online spend 70% of their time searching for information.
85-90% of all Internet users rely on search engines to locate sites.
Less than 10% of Internet users look past the first three pages of search results.

What all this means ?

If you want to have a hugely successful web site, you should do your best to get into the first three pages of as many search engines as possible. This article is about understanding search engines optimization so that you can better utilize your time to reach this goal.

Let's go to analyze how search engines are built. Search engines have three major components:

The first is a software program, called with a creepy name like spider or robot, which crawl around the web and visit web pages. These robots take that information to their home base: the database.
The database is something like a catalog and contains a copy of every single page that the robot scans. When a robot returns to your website and scans your website it will update the catalog. However, the reality is that in many search engines, the database does not index all the pages that the robot scans. Therefore, if you see in your tracking logs that a robot has visited and requested certain pages, this doesn't mean that those pages have been indexed. It is best to search the search engine for your URLs in order to verify if they are indexed or not.
The third component is the Ranking mechanism. This is the algorithm that matches the web surfer's keywords with the pages in the database and comes up with matches. These matches are then displayed on the surfer's screen. The higher you are displayed in the surfer's window, the better your chances are of being found.

These are the common denominators among the different major search engines. As the search engines are owned by different companies, their robots are tweaked differently.

Here is how they are tweaked:

1. The Robots

All the robots "deep crawl", which means that they crawl several layers into your website. Some search engines "Instant Indexes", which means that the information is put into the database of websites within days. All other search engines take longer so that you will need to wait after you have submitted your URL to the search engine.

Many search engines cannot crawl nor index frame pages. It is possible to circumvent the problems with framed pages of some search engines by using the <NOFRAMES> tag but it's better to avoid frames all together.

If you have an image map as one of your main pages for links to other areas of your website, you will have problems indexing your pages with some search engine robots: they cannot follow the links in an image map.

All robots will read your robot.txt file if present.

If you have Meta Robot Tags in your code not all search engines robots can read it. Google, for example, may not support checking it.

Work hard (and smart) to have as many well regarded websites linked to yours, known as Link Popularity. Link Popularity is extremely important because all search engines can determine how many links are going to and from your page. Some decide to "index or not" based on this attribute of your website. If you do this, then you can be rewarded by some search engines' robots deep crawling your website and therefore indexing more of your pages.

Many search engines, and the number is growing, see this with the rationale that if there are many links connected to a page, then it must be of more importance. This is becoming a popular tweaking among the programming of many robots.

2. The Database At The Home Base

All of the major search engines' databases will index the full body text. There may be certain words missing that are called "stop" words. These stop words are words that many robots skip in order to move faster. Not all robots will skip the same words of course and there are some that does not skip words.

Concerning your meta tags, all search engines index Meta Keywords except few. Additionally, all robots will index Meta Descriptions but the importance they are giving to this tags for ranking is decreasing.

If you have alt tags for your images on your pages that are being indexed then, some search engines will read and index them but others will not.

Any comment tags that you have in your web page code can only be read by very few robots.

3. Ranking

If you're hoping that your Meta Tags will boost your ranking for all search engines, this is not the case. Although all major search engines robots read and index meta tags, only few do give a boost to meta tags.

The important shift has been toward Link Popularity. As stated earlier, this is an important point to several search engines insofar as indexing is concerned. Additionally, it is also important in Ranking. Getting as many links of good sites linking to yours is an important point with all search engines.

Chose descriptive keywords to reflect the content. Have great content and a really good TITLE and Meta Keyword Description that entice people to click on your link when it comes up in a search, then you win points with a growing number of search engines.

The height of the boost depends on the number of people who click on your link.

One point to remember is that search engines, like most entities, change. The information you are reading is current now but changes in both, climate and search engine tweaking are always in the air.