A Mathematical Method with Word Frequency Sorting for Mining Web Content Outliers

The Internet is a massive collection of information that makes it extremely difficult to search and retrieve the required and valuable information. So, Search engine became an important tool for searching various data from the web. The primary evaluation of search engine is effectiveness and efficiency. While searching for information through search engines, always users retrieve redundant and irrelevant information. This replicated and uninteresting information affects both the effectiveness and efficiency of search engine by wasting users’ time by browsing the uninterested documents and its accessing time. Web content outlier mining plays a significant role in identifying and removing these redundant document (outliers) which is an important issue among the information retrieval and web mining research communities since most of the people rely on search engines to get the required information. Most existing algorithms for web content outlier mining focuses attention on applying weightage only to the common terms in the documents whereas in this research work, a mathematical approach based on term frequency ranking to identify the duplicates, and uses the domain dictionary to check for relevant document has been carried out to improve the effectiveness and efficiency of the search engine.