Pompeu Fabra University (UPF) is to lead a European research project to make a huge leap forward in how we interact with virtual environments and revolutionize the frontiers between the physical and virtual worlds. The project titled Research on Extended And Layered Multimodalities (REALM) aims to generate virtual experiences that are accessible from different devices so that viewers can immerse themselves in the story on their computer or tablet or wearing 3D glasses.

The three-year project aims for digital immersion to be not just an isolated visual experience but a multimedia continuum present in our daily lives, regardless of the device we use. To this end, it will receive 6.2 million euros from the EU Horizon programme, from the “Core technologies for virtual worlds” call.

UPF will lead the European consortium, which includes 10 partners from the academic world and leading companies in the field of media and entertainment. The consortium coordinators are the researchers Boris Bellalta and Ricardo Marques of the Department of Engineering where the former coordinates the research group on wireless networks and the latter is dedicated to interactive technologies.

Boris Bellalta, director of the UPF Wireless Networking research group, explains:  “With REALM, we aim to develop technologies to improve the user’s sense of immersion in virtual environments to provide as realistic multimodal experiences as possible, where key aspects such as vision, sound and touch, among others, are combined in a synchronized way”.

Ricardo Marques, of the Graphics and Interactive Technologies (GTI) research group, adds:  “We will develop new photorealistic image synthesis solutions based on lightweight machine learning models to give rise to more realistic visual experiences within easily accessible cross-platform web environments”.

Designing user-centred interactive virtual experiences

Unlike other purely technological approaches, REALM is based on a people-centred methodology. The project will be deployed through three main axes:

• Developing new image capture technologies: the aim is to create technologies that can not only record images, but also data related to textures, temperatures and surfaces that capture the complexity of the real world.
• Generating tools to create transformative virtual environments: from the data captured using the aforementioned technologies, these tools would be able to generate realistic virtual environments with which users could easily interact.
• Creating virtual content distribution platforms focusing on people’s needs: generating a shared framework between different devices to facilitate media convergence and the creation of cross-media virtual worlds. The platforms would be based on advanced technologies such as generative AI, volumetric reconstruction, reliable and low-latency (with few delays) communication networks, and haptics (the study of perceptions through touch).

A project that seeks to have an impact beyond entertainment: virtual experiences applicable to industry and healthcare

Although the creative industries will be the first to benefit, REALM is conceived with a clear vocation for knowledge transfer and it is hoped that some of its advances may be applied to industry, both in the manufacture of products and in the training of personnel, or in the field of healthcare.

The REALM Consortium

The European consortium that will develop the REALM project is made up of research centres and leading companies and organizations in optics, photonics, 3D graphics, networking, haptics (touch perception studies), and immersive arts. Below, the partners of the consortium are detailed by type.

Universities

• Pompeu Fabra University (Coordinator)
• KU Leuven (Belgium)
• University of Patras (Greece)
• Lucerne University of Applied Sciences and Arts (Switzerland)

Media Technology Suppliers

• Cooke Optics (UK)
• Brainstorm Multimedia (Spain)
• EVS Broadcast Systems (Portugal-Belgium)
• FilmLight GmbH (Germany)
• Alfalite (Spain)

Immersive art performance company

• CREW (Belgium)