Rookognise: Acoustic detection and identification of individual rooks in field recordings using multi-task neural networks

Individual-level monitoring is essential in many behavioural and bioacoustics studies. Collecting and annotating those data is costly in terms of human effort, but necessary prior to conducting analysis. In particular, many studies on bird vocalisations also involve manipulating the animals or human presence during observations, which may bias vocal production. Autonomous recording units can be used to collect large amounts of data without human supervision, largely removing those sources of bias. Deep learning can further facilitate the annotation of large amounts of data, for instance to detect vocalisations, identify the species, or recognise the vocalisation types in recordings. Acoustic individual identification, however, has so far largely remained limited to a single vocalisation type for a given species. This has limited the use of those techniques for automated data collection on raw recordings, where many individuals can produce vocalisations of varying complexity, poten-tially overlapping one another, with the additional presence of unknown and varying background noise. This paper aims at bridging this gap by developing a system to identify individual animals in those difficult condi-tions. Our system leverages a combination of multi-scale information integration, multi-channel audio and multi-task learning. The multi-task learning paradigm is based the overall task into four sub-tasks, three of which are auxiliary tasks: the detection and segmentation of vocalisations against other noises, the classification of in-dividuals vocalising at any point during a sample, and the sexing of detected vocalisations. The fourth task is the overall identification of individuals. To test our approach, we recorded a captive group of rooks, a Eurasian social corvid with a diverse vocal repertoire. We used a multi-microphone array and collected a large scale dataset of time-stamped and identified vocalisations recorded, and found the system to work reliably for the defined tasks. To our knowledge, the system is the first to acoustically identify individuals regardless of the vocalisation produced. Our system can readily assist data collection and individual monitoring of groups of animals in both outdoor and indoor settings, even across long periods of time, and regardless of a species' vocal complexity. All data and code used in this article is available online.