Testing Accuracy and Agreement among Multiple Versions of Automated Bat Call Classification Software

Passive acoustic monitoring is a common method of studying bats that involves recording echolocation calls of bats in their natural environment. Call sequences are then identified to species using automated acoustic analysis software. One limitation of acoustic software programs, particularly for long-term monitoring efforts, is that newer versions may provide results that are not directly comparable with older versions. However, there is little available information regarding how much or in what ways the versions differ, or which versions are most accurate. We evaluated 2 software programs used for automated bat call identification by testing a common set of echolocation call files across multiple versions of each program. We quantified the level of agreement on identification results and compared accuracy rates among the versions of each program. Level of agreement varied by species, recording location, and the software versions being compared. Overall percent agreement ranged from 28-82%. Newer versions were more conservative, in that they assigned fewer species-level identifications. However, newer versions were not substantially more accurate than older versions. Our conclusions suggest that bat researchers should be attentive to what software versions and settings are used as they plan and perform data analyses. Software developers could assist software users by providing more detailed information about their testing procedures and results, and what changes are associated with new versions. (c) 2021 The Wildlife Society.