How search engines work

Most likely, you have probably used a Web search engine such as Google, Yahoo, AltaVista or MSN to find specific information on the Internet. Did you ever stop and wonder how exactly do search engines find that information?
I will explain that and more in this section. In so doing, you will begin to discover how to efficiently structure your web site to obtain maximum brand strength, utilizing the power of today's modern search engines.
First, let's eliminate the mythsIt's a very common misconception that when a user enters a query into any search engine, it interrogates the Web to find pages that match the query. That is NOT how it works at all. Instead, the search engine looks at its own copy of the Web. Every search engine actually creates its own version of the Internet. This version is called an "index".
The size of a search engine's index varies from search engine to search engine, but it is always much smaller than the Web as a whole. For example, as of February 28, 2004, it is currently estimated that the whole Web presently consists of approximately 15 to 20 billion pages, whereas Google, which has the largest search engine index, has approximately 6 billion pages in it's index.
In fact, as late as October 2003, Google had only about 3.3 billion pages in its index, according to information available on its homepage at that time.
The search engine builds a list of pages to add to its index using a special piece of software known as a crawler or spider. The spider crawls across the Web, adding pages it visits to the list of pages to its index.
The spider is capable of reading text on a Web page and finding links to other pages to visit. In this fashion, the spider travels all across the Web, constantly finding new or modified pages to add or update to its current index.
Some time after a page has been "spidered" (visited by a crawler), the search engine's software effectively adds a copy of new or altered pages to the search engine's index. When a user enters a query into a search engine, the search engine's software searches its index to find the pages matching the search query.
It then sorts those pages into a specific ranking order. Each search engine uses its own search algorithm to find and rank pages, but most base their technology on the frequency and location of the search term on the page.
Apart from Google's Page Rank™ algorithm, engineers at Google have also developed the Hilltop™ algo, which is even more sophisticated.
The Hilltop algorithm determines the relevance and importance of a specific web page, determined by the search query or keyword used in the search box. In its basic, simplest form, instead of relying only on the PageRank™ value to find “authoritative pages”, it would be more useful if that “PR value” (PageRank™ value) would be more relevant by the topic or subject of that same page. Hilltop does that, plus a bit more.