Wordchillies » latent semantic indexing http://www.papertip.com/blog Letters & Words! Fri, 06 Aug 2010 10:42:26 +0000 http://wordpress.org/?v=2.9.2 en hourly 1 LSI – The Search Engine Think-Tank http://www.papertip.com/blog/lsi-%e2%80%93-the-search-engine-think-tank.html http://www.papertip.com/blog/lsi-%e2%80%93-the-search-engine-think-tank.html#comments Fri, 06 Nov 2009 06:07:01 +0000 Adam http://www.papertip.com/blog/?p=184 web1-web4Latent Semantic Analysis is primarily used by Google to detect spam, where excessive use of keyword is inserted in order to fool the search engines into providing a higher page-ranking for that keyword. This was indeed achievable by writing meaningless templates with rotating synonyms in which any keyword was multiply inserted by using software. Thousands of pages could be generated each hour targeting a specific keyword, and people were making thousands each day using Adsense.

The principles of LSI in determining content of web-pages, was first used by a small company called Oingo. Later renamed Applied Semantics, they developed a search system for determining the relevance of page content for specific advert placement, and called this Adsense. This was bought by Google in April, 2003 and used for replacing their system which was still developing. Adsense was not developed by Google, but purchased, and a good bargain at that.

However, a general opinion of the LSI is slightly off-mark. Instead of thinking it to be just an algorithm screwed on to the search engine, this could be better understood as a concept. Tie in the phrase ‘Artificial Intelligence’ to the LSI, and we see this in better light.

When we search for Tiger Woods, the search will not result in putting forth pages related to the keywords ‘tiger’ and ‘woods’; instead what we get is what we call ‘relevance feedback.’  This search will present a number of pages relating to golf. LSI is the most potent tool available for the search engines, as of now.

The LSI enabled search platforms are most effective and able to make better sense since they do not just focus on a bunch of keywords. Conventional search engines were unable to give good results since they were basing results only on ‘keywords’ analysis.

This was the reason they were unable to tell the difference between:

  • Similar words but with different meanings – and we have a whole lot of them.
  • Words similar in meaning, but spelled differently – sickness/vomiting.
  • Singular and plural forms of words – dice/die, man/men.
  • Branches of words from the same root – like ‘bath’, ‘bathe’, ‘bathing’, or ‘bathed.’

LSI is different, a kind of ‘Artificial Intelligence’ really. Looking at the speed and number of web-pages being added on the net everyday, it definitely is a challenging task for any search engine to provide an ideal search result.

The fact is that LSI is able to fit in this slot of search process, widely & wisely capable of enhancing the qualities of any search engine using LSI.

]]> http://www.papertip.com/blog/lsi-%e2%80%93-the-search-engine-think-tank.html/feed 1