Comparison of Different Ranking Methods in Wine Tasting

In this paper, we compare three ranking methods in wine tasting in terms of their respective accuracy levels. The first two are the original-score average and rank average, which are conventional methods in practice. The third is a relatively new ranking method called Shapley ranking. It is a game-theory-based ranking method, whereby judges are required not to rank order or score all the wines but only to choose a subset that they find meritorious. A simulation study is designed, wherein the data-generating scheme mimics how the real wine-tasting data are produced. We also consider two criteria in the comparison: the squared-error loss, which is a suitable measure when accurate ranking of all wines is of interest; and the percentile loss, which only considers whether the wines are correctly put in a certain subset. The main conclusion from our study is that the ranking based on score average is generally more accurate than that based on rank average. Shapley ranking, with the consideration that it puts less burden on judges in wine tasting, may outperform the other methods in certain conditions.