How to evaluate rankings of academic entities using test data.

•Framework for computing significance values when ranking algorithms are compared.•We analyse the stability and discriminative power of common evaluation measures.•Average and median rank have high discriminative power and are stable measures.•The nDCG measure only performs well when used on permille rankings.