WSG Newsletter: Web Search Report

Issue: July 16, 2001

Quietly, our favourite Web search tools have been changing. The dot-com crisis has made a dent at big portals like Altavista, NBCi, and Excite. Canada, especially, has taken some bad knocks. But not all is bad news. Google keeps getting better; there is Vivisimo to be thankful for and also Profusion. In Canada, Sympatico has been getting stronger. This newsletter looks at change in Web search tools over the past five to six months. Fasten your seat belts.

Subject Directories

Yahoo, Looksmart, and Open Directory Project rule. But are they impartial? Yahoo and Looksmart are deeply into paid-placement programs. Looksmart has gone one step further and introduced “subsite listings” (for a fee of course): it will add to subject categories pages with product information. A search for Elmore Leonard will find pages in Amazon about his books and list them as “reviewed web sites”. Looksmart is used at Excite, MSN, Altavista, Netscape, and Go2Net - all will be displaying the subsite listings. We can no longer count on subject directories to stick to sites.

Open Directory Project is still non-commercial and free to any web site that wants to tap in (and hundreds do). But it might get some competition from JoeAnt.com staffed mainly by refugees from Go.com’s Guide directory (now very dead), again on the volunteer basis. JoeAnt has a long way to go. (First spiders, now ants, when praying mantis?)

Britannica.com is the sorrowful news. Britannica, highly regarded for online research, is gradually changing to for-fee access. By the end of July, free content at the Britannica.com site will be much reduced consisting mainly of Britannica’s Internet Guide eBIG, perhaps articles from a collection of 70 some magazines, and lead paragraphs from articles in the encyclopedia.

eBIG used to be a superior directory of rated and reviewed sites. But browsing the guide’s subject tree has been removed and only the keyword search remains. Content is a little dated – Britannica still refers to guides at the Mining Company, which was renamed at least two years ago to About.com. With all the cuts in staff one wonders if eBIG is maintained at all.

Britannica’s retreat as a subject directory may open the field for high quality public library sites; sites like Librarians Index to the Internet (now added to TIG), Internet Public Library (a long-time TIG favourite), or Toronto Public Library’s Virtual Library.

Search Engines

Google is now handling 100 million searches a day. In the last four months it has added an image search (images.google.com), and translation of several languages to English (check preferences). TIG’s newsletter in August will be all about Google.

While Google makes steady progress, AltaVista is nearly paroxysmal. It did succeed in returning to its search roots with a new home page, and re-introduced a forms-based Search Assistant. Raging Search, once promoted as fast and simple, was dropped. Altavista added paid-for “featured sites” to its results page. It discontinued a directory search, but for hits it gets from its version of the Looksmart directory it will show the associated Topic.

Most significantly AltaVista has been erratic in changes to syntax. At present AltaVista is back to being an OR engine – it looks for any of the search terms. The Simple Search now accepts the boolean operators IF they are in upper case: AND, OR, AND NOT, NEAR. Advanced Search is still only boolean and will accept lower case. Simple Search is no longer case sensitive but the Advanced Search is.

Excite is still holding on although it did close many of its European portals. Excite Search has improved. The database has grown to 330 million, just trailing Northern Light’s 350 million. ZoomIn can propose alternate spellings and words for a search query. Excite will now group all pages from a site into a single listing – click on More from this site to get more. Unlike AltaVista, Excite kept its directory search. Excite also converted Webcrawler into pure search. It is Excite without the portal paraphernalia.

AlltheWeb (or FAST) (www.alltheweb.com) had a makeover that should make it a top choice for searchers. It too can group hits from the same site – they call it “site collapsing”. The Advanced Search Form offers many options for narrowing the search to title, domain, url etc. Search Tips is a new feature that will pop up depending on the search. Fast will also advise if there are any multimedia results. And it is still very fast.

MSN Search (search.msn.com) is a nice alternative to HotBot for searching the Inktomi database. Display is much less cluttered than it used to be. MSN will show articles from the Encarta encyclopedia, automatically correct spelling, and propose ways to broaden the search. Worth a look.

Paid Inclusion and Paid Placement

All the search engines have adopted paid inclusion programs whereby buyers are guaranteed to have pages at their web sites indexed and revisited periodically.

Many also have paid placement programs. For a charge per click, a web site will be listed among the top three or five on searches with matching keywords. Some have named this Featured Sites (Altavista, Hotbot, Lycos); others call them, more correctly, Sponsored Links (Excite). Altavista has its own program but also pulls in results from the best known of the pay4placement search engines, GoTo, as does Lycos, Hotbot and several others.

GoTo, which supplements its paid sites with hits from Inktomi, is the search engine now employed exclusively at Go.com and NBCi.com.

These aren’t necessarily bad and can be quite useful for searches such as travel where commercial sites are needed. However, they can overwhelm the results at some meta-search engines. As Danny Sullivan asked: meta search or meta ads? At Dogpile, a popular metasearch engine, he found, that 9 of the 15 search engines were paid-placement, and that 86% of the results were paid listings.

See Meta Search, Meta Ads? by Danny Sullivan. (May 23, 2001)
http://www.searchenginewatch.com/
sereport/01/05-metasearch.html

In Canada

Canada lost two search engines and gained one – sort of. In late June Altavista announced that it would assume full control of Altavista Canada. This ended a three-year partnership with Telus and all things Canadian about altavista.ca. Altavista Canada had been the only search engine to seek out Web servers in Canada and .ca sites outside of Canada. It had more Canadian pages than Altavista World. Today Altavista Canada only knows a Canadian site by the .ca domain. It won’t find Canada Computes or Information Highways magazine – anything in the .com, .net, .org domains. Also dropped were the Canadian News indexes, Canadian Government, and Health. It’s a great loss.

Canada.com is no better. It dropped Inktomi as the database and replaced it with Dogpile, a mediocre meta-search engine, heavy with paid listings, and not one bit Canadian.

The good news was that Google opened a Canadian version (www.google.ca) which does pick up Canadian content in domains other than .ca. It isn’t perfect and seems to include sites that are registered in the US and unrelated to Canada. Nonetheless, for access to indexed Canadian pages, Google.ca is the currently the best

Canadian (.ca) subject directories and search engines are listed with comments in TIG. See the pages in Web Searching for Best Subject Directories and Best Search Engines.

By far the best of Canadian sources today is Sympatico (www1.sympatico.ca). The agreement Sympatico has with Lycos has added to the portal web-based email, instant messenger, use of the Fast search engine, Direct Hit, and Open Directory Project. Sympatico also has an online bank – Amicus Financial and keeps adding Canada-specific content, such as the Government and Orgs section.

Meta Search Engines

The three most interesting meta-search engines are Vivisimo, Surfwax, and Profusion.com.

Yahoo! Internet Life named Vivisimo as the “best new search service” in the July 2001 issue, stating that Vivisimo has an “uncanny ability to track down what you're looking for”. Vivisimo does this by clustering pages into groups according to its analysis of the content – url, title, short description. It pulls results from several major search engines including Yahoo (with Google), MSN, Fast, AltaVista, Excite. It has also added meta-searches for news, and can be used on Pubmed, FirstGov (US government sources), and others. See http://www.vivisimo.com/demos for full list.

Surfwax has become even more a researcher’s tool. Display is very compact, with titles of results presented in the left frame according to relevance or alphabetically. More distinctively, SurfWax will create a SiteSnap for a page, essentially a snapshot of key points, words matched in context, and list of focus words which might be used as additional search terms. Most recently Surfwax has added advanced features for registered users. These include the ability to save SiteSnaps in an InfoCubby for use later, and to build SearchSets of tools from a choice of 1,200 search engines. Registration is free and includes the use of one SearchSet of the major search engines. There is a small annual fee to set up SearchSets to search the other search engines.

Intelliseek has made Profusion.com into the best meta-searcher of “vertical search engines” on the Web. Over 1,000 search engines are organized into 200 search groups. Select the group and then the search engines. Set up a search alert to be notified of new results for this search, or a page-watch to learn of changes at a specific page.

Conclusion

Web search has had its share of roadkill – Go.com, NBCi.com, AltaVista Canada come to mind, and some like Excite and Ask Jeeves have suffered. But there have been some new bright lights – Vivisimo and Surfwax are two, and several search services that have become better – MSN Search, Google, Fast, Sympatico.

Newsletter by Gwen Harris who is always looking for things.