A content-based technique for linking dual language news articles in an archive

To retrieve a specific news article from a vast archive containing multilingual news articles against a user query or based on similarity among news articles is a challenging task. The task becomes even further complicated when the archive contains articles from a low resourced and morphologically complex language like Urdu, along with English new articles. The article proposes a content-based (lexical) similarity measure, that is, Common Ratio Measure for Dual Language (CRMDL), for linking digital news articles published in various online news sources. The similarity measure links Urdu-to-English news articles during the preservation process using an Urdu-to-English lexicon. A literature review showed that an Urdu-to-English lexicon did not exist, and therefore, the first task was to build a lexicon from multiple sources. The proposed similarity measure, that is, CRMDL, is evaluated rigorously on different data sets, of varying sizes, to assess the effectiveness. The experimental results show that the proposed measure is feasible and effective for similarity computation between Urdu and English news articles, which can obtain, on average, 50% precision and 67% recall. The performance can be improved sufficiently by managing the limitations summarised in the study.