On the influence of the number of algorithms, problems, and independent runs in the comparison of evolutionary algorithms.

Graphical abstractDisplay Omitted HighlightsNHST and CRS4EAs have been compared with respect to k, N, and n.Both methods give similar conclusions regarding different numbers of algorithms k.The value of number of problems N affects NHST more than CRS4EAs.When the number of independent runs n is small, CRS4EAs is more reliable than NHST. When conducting a comparison between multiple algorithms on multiple optimisation problems it is expected that the number of algorithms, problems and even the number of independent runs will affect the final conclusions. Our question in this research was to what extent do these three factors affect the conclusions of standard Null Hypothesis Significance Testing (NHST) and the conclusions of our novel method for comparison and ranking the Chess Rating System for Evolutionary Algorithms (CRS4EAs). An extensive experiment was conducted and the results were gathered and saved of k=16 algorithms on N=40 optimisation problems over n=100 runs. These results were then analysed in a way that shows how these three values affect the final results, how they affect ranking and which values provide unreliable results. The influence of the number of algorithms was examined for values k={4, 8, 12, 16}, number of problems for values N={5, 10, 20, 40}, and number of independent runs for values n={10, 30, 50, 100}. We were also interested in the comparison between both methods NHST's Friedman test with post-hoc Nemenyi test and CRS4EAs to see if one of them has advantages over the other. Whilst the conclusions after analysing the values of k were pretty similar, this research showed that the wrong value of N can give unreliable results when analysing with the Friedman test. The Friedman test does not detect any or detects only a small number of significant differences for small values of N and the CRS4EAs does not have a problem with that. We have also shown that CRS4EAs is an appropriate method when only a small number of independent runs n are available.