Search Engines

Word Indexing

Search engines index the contents of pages on the World Wide Web. They are similar to indexes at the back of a book where words are listed with the page numbers where they occur: find the word and then check the page.

There are many search engines on the Web. They do basically the same thing:

1. Send out a software agent (called a spider, robot, crawler) to roam the Web, and follow links to new pages.
2. Use the spider to gather words from the pages.
3. Create a large database of words with each word and phrase indexed to its web address (or URL).
4. Provide a query form for you to use to enter your search words.
5. Query the database using particular search and retrieval logic.
6. Display the search results, usually with your search terms bolded in excerpts of the text.

However, they all do this differently. They:

1. Go out on the internet at different intervals.
2. Go to different sites.
3. Index different parts of the site.
4. Index more or less of the pages on the site.
5. Index different document types: web pages, Adobe Acrobat pages (pdf), PowerPoint files (ppt), Word documents (doc) ,Macromedia flash (swf). (Although most search engines index all of these types now.)
6. Have different search forms and search rules. Some can penetrate to searchable databases.
7. Rank results differently. The top 10 at Yahoo may not be the same as the top 10 at Google or Bing.
8. Display the results of a search differently.

In view of these differences, it is always wise to check two or three search engines for your search.

Note the difference from the Subject Directory: The Search Engine lets you search the words on web pages. The Subject Directory groups sites (and sometimes pages) according to what they are about. If your interest is in racehorses, a directory will provide a list of sites about racehorses; but, if you want to know about the famous Canadian thoroughbred Northern Dancer, a search engine will find pages where he is mentioned.

How to Best Use a Search Engine

Since these are very large databases of indexed words, you must be very specific in your search question. Think of as many unique terms as possible to describe what you are looking for. The more words you use the more likely you will be rewarded with relevant finds.

Usually a search involves an iterative cycle of entering some words, evaluating results, picking up some new words and trying them in a new search until you find your answer or a good set of relevant pages. You may need to take three or four cracks at a search.

Where to next?

Search Google in this Search Engine Exercise.