Roc Speak: Semi-Automated Personalized Feedback On Nonverbal Behavior From Recorded Videos

We present a framework that couples computer algorithms with human intelligence in order to automatically sense and interpret nonverbal behavior. The framework is cloudenabled and ubiquitously available via a web browser, and has been validated in the context of public speaking. The system automatically captures audio and video data in-browser through the user's webcam, and then analyzes the data for smiles, movement, and volume modulation. Our framework allows users to opt in and receive subjective feedback from Mechanical Turk workers ("Turkers"). Our system synthesizes the Turkers' interpretations, ratings, and comment rankings with the machine-sensed data and enables users to interact with, explore, and visualize personalized and presentational feedback. Our results provide quantitative and qualitative evidence in support of our proposed synthesized feedback, relative to video-only playback with impersonal tips. Our interface can be seen here: http://tinyurl.com/feedback-ui(Supported in Google Chrome.)