Internet News: Semantic Search Explained

July 30, 2009

Semantic Search Explained

Search: The Last Frontier by Barbara Brynko, Information Today (June 2009) - via AllBusiness.com

Report from the 2 day Infonortics Conference in Boston in April 2009. This is always cutting edge. Semantic search was the main topic.

Why Semantic Search?

Since searchers have begun wading through the quagmire of information, their needs have changed and so have their tolerance levels. There are many times when ? age -ranking results just don't produce what users are searching for on the web. Dmitri Soubbotin from Semantic Engines elaborated on three reasons users need semantic search. First, he says users deal with insufficient relevance of traditional search results; users just spend too much time searching for information but not always finding what they want. Second, users are pressed for time and have short attention spans; users want relevant information retrieved quickly. Third, most users only look at the first page of the results and don't even peek at the useful sources beyond. Far too many users say they will "settle for what I have here," he says.

But what's really under the hood? Instead of using ranking algorithms as Google does to try to predict relevancy for the user, semantic search uses the science of the meanings in language to produce point- on results. Natural language processing, linguistics, and text mining can be matched against an ontology that works especially well for verticals. Homogeneous content yields better results; there's just "less noise" and less disambiguation for users to deal with.

After all, the goal of the web is to extract more relevant results and to retrieve accurate answers for users while discovering additional content and digging deeper for pertinent data.

A search engine such as Sensebot provides an overview of a topic's hard facts interspersed in text results. Users receive a multidocument summary and links that go beyond simple information search and retrieval.

For Diane Burley of Nstein, "Search is so yesterday. ... It's now all about the finding."

But to make the process of finding information easier, we need to take a look at how people seek information, how they orient themselves, and what their sources of frustration may be, she says.

"Until users are inconvenienced, they don't see the value in the search process," she says. If concepts and entities are extracted, links give users more reason to stay on a site and make it easier for them to mine and to aggregate results, even across different languages and country borders.

Bringing Clarity to the Mix

For semantic search to work effectively, users need to maximize relevance and minimize disambiguation. Kathleen Dahlgren from Cognition Technologies explored approaches to tagging, ontology, syntax parsing, and a semantic map. The most common words are the most ambiguous, she says, using the word "lemon" as an example. A word string for "lemon" produces a number of possible definitions: It could be a citrus fruit, a poorly manufactured item, a yellow color property, or a behavioral property.

But word definitions are just part of the puzzle for semantic technology. Add concepts to the mix (lemons are typically yellow) and personal ignorance (pythons are dangerous, but what are pythons anyway?) and social ignorance (the sun revolves around the Earth), and users have the beginnings of a deeper search. In semantic analysis, the word is not only defined by its relevancy, it also takes into account the other words that are present in the sentence and as part of the context of the complete text. Less disambiguation means more-relevant results and a better understanding for the user.