Web Data Harvesting For Speech Understanding Grammar Induction

The development of a speech understanding grammar for spoken dialogue systems can be greatly accelerated by using an in-domain corpus. The development of such a corpus, however, is a slow and expensive process. This paper proposes unsupervised, language-agnostic methods for finding relevant corpora in the web and mining the most informative parts. We show that by utilizing perplexity we are able to increase the in-domainess (precision) of the mined corpora, while by utilizing pragmatic constraints and search engine rank we can increase the generalizability (recall). We show that automatic grammar induction algorithms achieve superior performance on the automatically mined corpora compared to in-domain manually collected corpora for a travel application.