A human-in-the-loop recommendation-based framework for reconstruction of mechanically shre dde d documents

The advances in machine learning - particularly in deep learning - have enabled automatizing the re-construction of shredded documents with significant accuracy. However, despite the recent remarkable results, the state-of-the-art on fully automatic reconstruction still has room for improvement, mainly due to imprecision on the evaluation of how the shreds fit each other (compatibility/cost evaluation). To tackle this problem, we propose a human-in-the-loop reconstruction framework that takes user inputs to im-prove the solutions (permutation of shreds). In our approach, the user verifies whether adjacent shreds of a solution are also adjacent in the original document. Unlike the current literature, our framework includes a recommender module that automatically selects pairs of shreds to be analyzed by a human. Four recommendation strategies were proposed and evaluated. Results achieved by coupling deep learn-ing reconstruction methods into our framework have shown that introducing the human in the loop can reduce errors by more than 40% .(c) 2022 Elsevier B.V. All rights reserved.