Inter-rater agreement and adjusted overall degree of distinguishability for ordered categories

In square contingency tables, weighted kappa and AC2 coefficients are used to summarize the degree of agreement between raters of an ordered square contingency table. In addition to investigate the agreement between raters, category distinguishability should be considered to check the reliability of the study. The overall degree of distinguishability is used for RxR tables. In some applications, the value of overall degree of distinguishability is calculated outside the defined range as negative values. Since overall degree of distinguishability is calculated by using all the category pairs, there occurs inflation on its value. In this study, adjusted overall degree of distinguishability is suggested to solve these two problems. Furthermore, interpretation of category distinguishability is outlined and benchmark scales for overall degree of distinguishability are developed. A simulation study is performed to compare the accuracy of the adjusted overall degree of distinguishability and the classical one. The tables to find the adjusted overall degrees of distinguishability for certain values of agreement coefficients are generated. The results are discussed over medical data sets.