Observing Pianist Accuracy And Form With Computer Vision
2019 IEEE WINTER CONFERENCE ON APPLICATIONS OF COMPUTER VISION (WACV)(2019)
摘要
We present a first step towards developing an interactive piano tutoring system that can observe a student playing the piano and give feedback about hand movements and musical accuracy. In particular, we have two primary aims: (1) to determine which notes on a piano are being played at any moment in time, (2) to identify which finger is pressing each note. We introduce a novel two-stream convolutional neural network that takes video and audio inputs together for detecting pressed notes and fingerings. We formulate our two problems in terms of multi-task learning and extend a state-of-the-art object detection model to incorporate both audio and visual features. We also introduce a technique for identifying fingerings if pressed piano keys are already known. We evaluate our techniques on a new dataset of multiple people playing several pieces of different difficulties on an ordinary piano.
更多查看译文
关键词
two-stream convolutional neural network,audio inputs,pressed notes,multitask learning,audio features,visual features,pressed piano note information,pressed piano keys,piano player,pianist accuracy,computer vision,interactive piano tutoring system,object detection model,finger identification solution
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要