On applying metamorphic testing: an empirical study on academic search engines

Software testing can be a helpful practice to certify the quality of a product. However, there are programs which are hard, if not impossible, to determine the expected outputs. This problem is called the "oracle problem". Metamorphic testing (MT) is an approach that aims to alleviate this problem by setting a series of relations, called metamorphic relations (MRs). This paper applies MT through a user-oriented approach and the following MRs: MPublished, MPTitle, MPShuffleJD and Top1Absent as a strategy for evaluating, verifying, and validating four Academic Search Engines (ASEs): ACM, IEEE, ScienceDirect, and Springer. Therefore, we conducted an experimental study to analyze how MRs can contribute to verifying the correctness of the properties of ASEs. Results indicate that the ASEs have performed differently in their execution for each MR analyzed. This shows that the algorithms of the scientific search engines have different behaviors among the characteristics analyzed. These behaviors may not be properly characterized by a possible failure, but by a simple design choice. On the other hand, the approach is useful for search engine users that need to be sure that the ASEs behave as expected during a search, as the results of secondary studies strongly depend on the correct behavior of the scientific search engine used.