Towards open-world multimedia forensics through media signature encoding

Countering image and video manipulations is getting more and more relevant in several fields such as investigation, intelligence and forensics. Multimedia forensics researchers keep developing newtools and updating available detectors to discriminate the processing the media has been subjected to. Even though these tools can be effectively used under controlled environments, they are generally unreliable in open-world settings, where the investigated content may have undergone several unknown processing. In this paper, we present a novel framework to discriminate different toolchains of media manipulation and processing. We introduce the concept of media signature encoding to map image and video contents in spaces where media corresponding to similar processing toolchains cluster together. We also demonstrate that this property still holds for toolchain that are not known when building the encoder, thus making the framework applicable in open-world settings where the forensic practitioner can face both known and unknown manipulations. It is worth to note that an advantage of this approach is that in principle media signatures can be generated starting from any kind of forensic features. We tested the effectiveness of the proposed framework in two different experimental setups involving digital images and videos, respectively. Experiments show that encoded signatures are capable of determining whether: (i) a media under analysis belongs to a known life cycle or an entirely novel processing toolchain; (ii) a subset of media items share the same history. This framework can be considered a first step towards the use of forensic features to characterize media life cycles in open-world settings.