eDgE platforM fOr dynamiC xR applicATionS (DEMOCRATS)

Abstract:

Extended Reality (XR) applications with a new level of immersive experiences can reshape our society. Especially, the revolutionary Mixed Reality technology, providing novel ways of interactions among arbitrary physical and virtual entities, enables a new generation of applications in many fields. Medical and health services, Industry 5.0, online gaming, architecture are just highlighted examples with special importance and business potential. However, the required quality of user experience and the desired high visual fidelity pose several challenges. Simulation of the virtual 3D environment, real-time object/user detection and tracking, high fidelity 3D rendering are crucial CPU/GPU intensive tasks greatly affecting the immersive experience. Today’s (and near future’s) XR devices, even the high-end ones, such as Hololens 2 HMD (Head-Mounted Display), cannot provide the required high quality 3D models and essential XR functions due to limited CPU/GPU capacity. Offloading CPU/GPU intensive XR functions to the edge cloud infrastructure can resolve this issue together with 6G (or beyond 5G) systems which fulfill the related networking (latency and bandwidth) requirements.  

In this project, we conduct experiments with our novel edge/cloud-based XR platform providing essential features for future XR applications and we also analyze the user experience in a specific multi-user XR application built on the platform. More specifically, our novel split rendering framework (combining remote and local rendering) with occlusion support and dynamic environment detection is deployed to the Imagine B5G infrastructure and evaluated from multiple aspects making use of a real application in various scenarios. We highlight the advantages of split rendering and the challenges of occlusion support in dynamic environments. In addition, we showcase different solutions for occlusion support, reveal pros/cons of the approaches, analyze their feasibility in different application areas and propose tailor-made warp techniques for the specific B5G network environment which can hide the impact of dynamically changing transmission characteristics. 

Contacts: 

Balázs Sonkoly - sonkoly.balazs@vik.bme.hu

Bálint György Nagy - nagy.balint.gyorgy@vik.bme.hu

János Dóka - doka.janos@vik.bme.hu