An IoT-based System of Converting Handwritten Text to Editable Format via Gesture Recognition

Despite the prevalence of online learning or e-learning, the evolution of traditional classrooms has never stopped. Electronic classrooms, where classrooms equipped with projectors, computers, and sound systems, have become dominant in the last decade. The next step of evolution for electronic classrooms is smart classrooms. One of the most popular features of an electronic classroom is capturing video/photos of lecture content and extracting handwriting for note-taking. Various techniques have been implemented in order to extract handwriting from the video/photos of the lecture. However, there are still deficiencies in the existing techniques that prevent us from turning an electronic classroom into a smart classroom. This paper presents a real-time Internet of Things (loT)-based system to convert handwritten text into editable format by implementing Hand Gesture Recognition (HGR) with Raspberry Pi. Handwritten images are converted into editable format by using OpenCV and machine learning algorithms. For text conversion, the recognition of uppercase and lowercase alphabets, numbers, special characters, mathematical symbols, equations, graphs, and figures is included with the recognition of words, lines, blocks, and paragraphs. With the help of Raspberry Pi and IoT technologies, students can access the editable format of lecture notes via a desktop application, which helps students to edit and share notes and images according to their necessity. Implementation details and comprehensive evaluations of the system are summarized in the paper.