Pairwise statistical comparisons of multiple algorithms

In machine learning, it is often necessary to statistically compare the overall performance of two algorithms (e.g., our proposed algorithm and each compared baseline) based on multiple benchmark datasets. In this case, our proposed algorithm is typically referred to as the control algorithm. However, in some cases, it is also essential to conduct pairwise statistical comparisons of multiple algorithms without a control algorithm.

To conduct pairwise statistical comparisons, a research team led by Min-Ling ZHANG published their new research on 15 December 2025 in Frontiers of Computer Science co-published by Higher Education Press and Springer Nature.

The team proposed that the average rank-based strategy (e.g., the combination of the Friedman test and the Nemenyi post-hoc test) can lead to test results that are inconsistent with common sense. Moreover, they designed a new presentation plot to report the results of pairwise statistical comparisons.

In the work, they analyze that the value of critical difference (CD) is often greater than one in our daily machine learning research, then the combination of the Friedman test and the Nemenyi post-hoc test will usually lead to test results that are inconsistent with common sense. They recommend strategies that are not based on average ranking for statistical comparison (e.g., Wilcoxon signed-ranks test). To present the experimental results of pairwise statistical comparisons, they further design a new presentation plot based on equilateral polygon. Specifically, each vertex corresponds to one algorithm. If one algorithm achieves statistically better performance than another, an arrow is used to connect them from the better one to the other. Otherwise, a dashed line is used to connect them.

Future work can focus on applying the proposed presentation plot to more pairwise statistical comparisons in related researches.