April 16, 2002
Okay, so you have developed a strategy for search engine submission, fixed a budget for it and submitted your site to Web directories and crawlers. Is that enough for you to attract users to your site?
Well, maybe if your luck holds out like the proverbial luck of the Irish. But usually, doing only so much does not bring you the hordes of Web surfers that you need to make your business a roaring success.
To get hordes of surfers to visit your sites through the search engines, you need to make sure that your site ranks high enough within the searches that are relevant to your product offerings. Only then will those Web surfers notice your site and visit it, which in turn may necessitate some tweaking or tuning of the Web pages. And that requires that you understand how search engines work, what appeals to them and what changes you need to incorporate in your Web pages to improve their rankings within search results.
How Web Directories Work
Most Web directories, like Yahoo, are powered by humans and depend upon them to create the listings. For these directories, human editors review the URL submission--title, description, keywords etc.--and create the record in the database. A search through the Web directories looks for matches within the title, descriptions and other details maintained by the directory. So if you change your Web page, it would have no effect on the search results unless the accompanying record in the Web directory is also changed.
The bottom line for being effective with the human-powered Web directories is how you write the title, description and other details for the URL that you submitted for inclusion in the directory. Of course it also depends upon what the human editor thinks in terms of how your description relates to the content of your URL, and whether it adheres to the policies of the Web directory. Hence, what may be useful for improving a listing with a search engine may not help improve your chances of listing within a directory.
How Crawlers Work
As the name implies, a crawler-based search engine creates its listings automatically by crawling the website. Using software tools called bots, they "crawl" or "spider" the Web pages, cataloging or indexing the results. "Bot" is short for robot and is a software tool or agent that digs through data based upon the directions given to it and returns answers from the Web pages that it crawls. An example could be a shopping bot that compiles the enormous database or products sold at online stores.
Anything that these bots or spiders find goes in an index (also called the catalog) that contains a copy of every Web page that the spider finds. Usually the crawlers crawl through their database of submitted URLs periodically. Some do it every week and some every other week. As they do it periodically, they eventually find any changes that you make to your Web pages and update their indexes with new information. However, there may be a time lag for a Web page to be crawled and indexed, and the page is only available for a search when it has been indexed.
For these crawlers, page titles, body copy and all other elements play a part. One can imagine that since these are automated programs using artificial intelligence that they would skip parts of the Web pages that they do not understand. Different crawlers handle advanced features like image maps, dynamic page creation, etc., differently. Some can index them, and some may not. The bottom line is that to be effective with the crawlers you need to write Web pages that are spider-friendly.
Presenting the Results
Do all search engines work similarly? Does a search engine present results from only one type of source--crawler or web-directory? Well, not any more. Most search engines present mixed results, though it is possible that they may favor one over the other. Many top-notch search engines have relationships with various players to present different types of results. For example, at times Yahoo also presents results from Google along with the results from its own Web directory.
To bring the result of a query to the user, all search engines perform three types of functions. But how they do it differs between the types of search engines, and also within each type.
Capturing the URLs of the Web pages is the first activity. Web directories do it by the manual submission and crawlers do it through bots. The second activity is indexing the captured pages. In the case of Web directories, human editors create entries in the indices, whereas crawlers maintain their indices automatically.
The third activity is to find and present the information sought by the user. This part of the search engine software sifts through millions of pages recorded in the index to find matches to a search. This third activity is the most critical for you to attract hordes of Web surfers to your site--how a search engine ranks Web pages in the order that it believes is most relevant for the query.
This brings us to the next thing that the Web masters and website creators must know in order to improve the ranking of their pages within a search result--namely, how to create search engine friendly Web pages. But that is the topic of another article.
Strategic and results-oriented, Sunil has more than 15 years of experience in management and IT consulting. An entrepreneurial consultant, he had founded a business-to-business eCommerce company. Sunil has provided consulting services to large and small firms in the