An end-to-end distributed video analytics system using HEVC annotated regions SEI message

Deep learning-based workloads for the analysis of video data are on the rise. Examples include residential and commercial surveillance systems where the camera data are analyzed for potential intruders, in-door retail cameras that track people movement for behavior analysis, content curation etc. In an End-to-End (E2E) intelligent video solution, video analytics will be carried out at multiple places from low-power edge IP camera to high performance cloud servers. Typically, the results of the video analytics at one stage will be sent to the next stage along with the video data for efficient processing. We proposed a novel Annotated Regions Supplemental Enhancement Information (AR SEI) message to embed video analytics metadata within a compressed video bit stream that would help to embed analytics inferred at each stage of the pipeline along with the content. Our proposed SEI message has been included in the seventh edition of the High Efficiency Video Coding (HEVC) standard, will be included in upcoming editions of the Versatile Supplemental Enhancement Information (VSEI) and Advanced Video Coding (AVC) standards. In this paper, we illustrate the salient features of the AR SEI message and how it addresses & improvises End-to-End intelligent video use cases. We demonstrate features of the Gstreamer implementation of AR SEI handling in the context of both playback and end-to-end distributed video analytics pipelines.