LiTrix: A Lightweight Live Light Field Video Scheme for Metaverse Stereoscopic Applications

Aiming to bring 3D immersive and personalized experiences, the metaverse converges many emergent technologies, including Internet of Things (IoT)-based data collection, 5G networks, and Artificial Intelligence (AI)-assisted computer vision. One of the efforts in the metaverse supports light field video (LFV) collection and processing to create high-fidelity digital twins, which act as a fabric link between the physical world and the cyber world. However, the research on the LFV system is still hindered by complicated capturing systems and huge computational resource requirements. We propose LiTrix, including a lightweight device for capturing live LFV with real-world scenes, and the device-edge-cloud collaboration under 5G networks to conduct stereoscopic reconstruction with less consumption of resources. LiTrix is fabricated by leveraging IoT devices that include a portable 100-camera array and dedicated edge and cloud servers. A novel LFV dataset is recorded through the devices, providing a diversity of the texture of the scene varying from paper products to ceramic to cotton. The dataset is open source and shared for public usage. In addition, we present the results of point cloud and mesh reconstruction, and show that the IoT-based collaboration framework saves 19.6 percent of the time in the ablation experiment. We expect that LiTrix can be used for metaverse stereo-scopic applications in the future.