Evaluating participating methods in image analysis challenges: Lessons from MoNuSAC 2020.

Biomedical image analysis competitions often rank the participants based on a single metric that com-bines assessments of different aspects of the task at hand. While this is useful for declaring a single winner for a competition, it makes it difficult to assess the strengths and weaknesses of participating al-gorithms. By involving multiple capabilities (detection, segmentation and classification) and releasing the prediction masks provided by several teams, the MoNuSAC 2020 challenge provides an interesting op-portunity to look at what information may be lost by using entangled metrics. We analyse the challenge results based on the "Panoptic Quality" (PQ) used by the organizers, as well as on disentangled metrics that assess the detection, classification and segmentation abilities of the algorithms separately. We show that the PQ hides interesting aspects of the results, and that its sensitivity to small changes in the pre-diction masks makes it hard to interpret these results and to draw useful insights from them. Our results also demonstrate the necessity to have access, as much as possible, to the raw predictions provided by the participating teams so that challenge results can be more easily analysed and thus more useful to the research community.(c) 2023 Elsevier Ltd. All rights reserved.