Building Web-Based Subject-Specific Corpora on the Desktop: Evaluation of Search Metrics

Building subject-specific or domain corpora from Web data is well-researched. However, most approaches start by using seed articles as inputs to Web crawlers and take document similarity algorithms for selection. We take a different lean resource approach by applying traditional search metrics with a relatively large (more than 100 search terms) ‘bag of domain words’ approach on the colossal clean crawled corpus. This approach enables one to build rich domain corpora of text documents quickly in a resource-poor environment (e.g., a few CPU cores). This paper tests several metrics using three different subject domains—language, Colossal Clean Crawled Corpus basic mathematics, and information science—and finds that there are significant performance differences between the various metrics. Surprisingly, a naïve, simple metric, outperforms TD-IDF and performs almost as well as our top ranked algorithm, Okapi BM25. This demonstrates that the performance of search metrics using a relatively larger number of search key words (> 100) is different than when a small set of search key words is used. We also demonstrate how to optimize the free parameters for Okapi BM25.